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1.0  INTRODUCTION 


This  report  describes  the  first  case  study  performed  for  the  Combat  Automation  Requirements 
Testbed  (CART)  program.  CART  is  an  Air  Force  Research  Laboratory  Human  Effectiveness 
Directorate  program  that  is  developing  tools  that  strongly  support  simulation-based  acquisition 
(SB A)  concepts.  As  an  Advanced  Technology  Development  research  effort,  the  goal  of  CART 
is  to  demonstrate  and  evaluate  the  application  of  human  performance  modeling  to  the  design  and 
development  of  crew  systems  that  integrate  the  warfighter  and  weapon  system  more  effectively. 

The  demonstrations  performed  under  the  CART  program  occur  in  the  context  of  case  studies.  In 
a  case  study,  a  particular  human  performance  context  is  selected,  a  model  of  the  human 
performing  in  that  context  is  developed,  and  the  human  performance  model  is  integrated  with  a 
constructive  representation  of  the  system,  the  operator  controls,  and  the  mission  environment  in 
which  the  system  operates.  The  integrated  constructive  testbed  is  exercised  in  several  scenarios 
or  test  conditions  and  performance  data  are  obtained.  In  addition  to  constructive  testing,  virtual, 
human-in-the-loop  (HITL)  simulations  are  also  conducted  for  the  same  scenarios  and  test 
conditions.  Finally,  the  data  resulting  from  the  constructive  and  virtual  testing  are  compared  to 
determine  the  extent  to  which  the  human  performance  model  predicts  actual  human  performance. 

To  date,  the  first  of  two  CART  case  studies  has  been  completed  (Case  Study  1).  This  report 
provides  a  brief  overview  of  the  CART  program  and  then  describes  the  development,  conduct, 
and  results  of  Case  Study  1. 


1.1  The  Problem  CART  is  Addressing 

As  the  analytical  capabilities  and  potential  for  cost  savings  afforded  by  modeling  and  simulation 
(M&S)  technology  continue  to  expand,  the  width  and  breadth  of  M&S  applications  also  continue 
to  grow.  For  many  applications,  including  training,  analysis,  and  acquisition,  constructive 
simulations  of  systems  in  their  intended  environments  are  proving  extremely  valuable. 
Historically,  however,  one  limitation  of  such  constructive  simulation  environments  is  their 
ability  to  represent  the  human  component  of  the  manned  system  being  simulated.  While  we  are 
generally  quite  good  at  representing  performance  of  the  hardware  and  software  in  a  given 
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system,  we  tend  to  straggle  in  our  modeling  of  the  perceptual,  cognitive,  and  physical 
capabilities  of  the  operator  controlling  the  system.  As  a  result,  attributes  such  as  operator 
workload,  performance,  and  tactics  -  each  of  which  are  critical  components  of  overall  system 
performance  —  are  often  ignored  or  severely  constrained  within  the  constructive  simulation 
environment,  thereby  restricting  the  validity  and  generalizability  of  the  simulation  effort’s 
results.  The  Department  of  Defense  (DOD),  which  identified  “providing  authoritative 
representations  of  human  behavior”  as  one  of  six  key  goals  to  be  achieved  within  its  modeling 
and  simulation  efforts,  has  recognized  this  limitation  (DOD  5000.59-P). 

One  particular  application  of  modeling  and  simulation  in  which  the  human  representation  is 
often  lacking  is  the  area  of  acquisition.  Currently,  analysts  and  decision-makers  rely  heavily  on 
constructive  simulations  of  a  system  in  its  intended  environment  to  help  translate  mission 
requirements  identified  by  the  warfighter  into  system  performance  requirements.  Within 
constructive  simulations,  sensitivity  analyses  are  conducted  on  key  subsystem  attributes  by 
selectively  varying  attribute  levels  and  measuring  the  results  on  mission  performance.  In  this 
way,  performance  levels  are  identified  for  key  subsystem  attributes  that  yield  desired  levels  of 
mission  performance,  thereby  providing  the  basis  for  statements  of  system  requirements. 

Unfortunately,  consideration  of  the  crew  interface  as  part  of  the  system  is  generally  avoided  in 
these  requirements-generation  efforts.  The  acquisition  community  continues  to  use  expensive 
HITL  simulation,  in  part  because  of  its  inability  to  model  crewmember  behavior  and  human- 
computer  interactions  within  constructive  simulations.  Hence,  crew  interface  requirements  are 
not  quantifiably  linked  to  the  set  of  overarching  measures  of  weapon  system  effectiveness  as  the 
other  subsystem  attribute  requirements  are.  Not  only  can  the  lack  of  realistic  consideration  of  the 
operator  on  system-level  performance  requirements  lead  to  crew  interface  inadequacies,  it  can 
also  drive  performance  and  cost  unnecessarily  because  the  simulated  system  did  not  represent 
appropriate  tactics.  One  military  analyst  recently  noted,  “Every  single  analysis  that  I  have  ever 
seen  has  suffered  from  the  lack  of  capturing  smart  tactics.  Mistakes  such  as  pursuing  an  attack 
when  the  tactic  should  have  been  ‘ran  away’  lead  to  mission  outcomes  (aircraft  loss)  that  seem  to 
indicate  system  deficiencies  when  in  fact  the  system  was  misused  tactically.”  (Martin,  Brett  & 
Hoagland,  1999)  Analysts  and  decision-makers  need  a  means  to  readily  model  and  understand 
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the  effects  of  human  performance  on  total  weapon  system  effectiveness  when  translating 
operational  requirements  into  system  requirements,  and  they  need  to  be  able  to  visualize  these 
effects  at  different  levels  of  aggregation  (Martin,  et  al.,  1999). 

To  address  the  problem  outlined  above,  the  Air  Force  Research  Laboratory  initiated  the  CART 
program.  The  program’s  overall  objective  is  to  provide  a  tool  that  permits  users  of  constructive 
simulations  to  readily  develop  and  integrate  human  performance  models  in  an  effort  to  achieve 
more  accurate  representations  of  the  human  operator’s  impact  on  overall  mission  outcomes. 

More  specific  objectives  of  the  CART  program  are  to: 

(1)  advance  the  state-of-the-art  in  human  modeling  using  interoperable  simulations  and 
practices,  such  as  High-Level  Architecture  (HLA),  (Defense  Modeling  &  Simulation 
Office,  1998), 

(2)  demonstrate  a  robust  human  modeling  architecture  that  is  compatible  with  current 
and  future  DOD  simulations, 

(3)  link  operator  performance  with  mission  effectiveness,  and 

(4)  provide  the  capability  to  trace  cause-and-effect  relationships  during  or  after 
simulation  runs. 

1.2  CART  Program  Tools 

The  CART  program  will  extend  current  constructive  M&S  testbed  capabilities  by  providing  two 
new  tools  for  enhancing  human  performance  representations  in  constructive  simulation.  One  is  a 
human  performance  modeling  capability.  With  this  tool,  analysts  will  be  able  to  create  models 
that  simulate  activities  operators  would  perform  in  a  system.  Analysts  also  will  be  able  to  assign 
parameters  to  the  models  to  reflect  different  levels  of  operator  capability.  These  human 
performance  models  will  be  integrated  with  constructive  models  of  a  system  and  will  interact 
with  the  system  in  the  context  of  a  simulated  mission.  The  second  tool  will  provide  performance 
assessment  capabilities,  supporting  generation  of  measures  of  operator  performance  that  will  be 
clearly  linked  to  measures  of  system  performance  and  mission  effectiveness.  With  this  tool 
relationships  among  operator,  system  and  mission  performance  will  be  visualized  and  traced,  and 
levels  of  operator  performance  required  to  produce  desired  mission  outcomes  will  be  identified. 
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1.3  CART  Program’s  Phase  1  Components 

The  current  phase  of  the  CART  program  (Phase  1)  consists  of  six  tasks  that  support  the  program 
goals  and  development  of  the  CART  tools.  The  objectives  of  each  task  are  discussed  briefly 
below. 

Task  1:  Crew  System  Requirements  Establishment.  The  objective  of  this  task  is  to  characterize 
the  current  acquisition  process  for  DOD  acquisition  programs,  highlighting  the  nominal  extent  to 
which  crew  interface  requirements  are  analyzed  and  established.  Essentially,  the  end  goal  of 
Task  1  is  to  identify  the  ‘user’  of  CART,  the  environment  and  processes  in  use  today,  and  the 
most  appropriate  niche  for  CART  in  the  acquisition  environment. 

Task  2:  Human  Modeling  Architecture.  Task  2  focuses  on  defining  a  human  modeling 
architecture  to  be  integrated  with  a  selected  military  simulation.  It  involves  the  development  of  a 
human  performance  model  (HPM)  as  well  as  an  interface  to  an  engagement-level  simulation  that 
both  characterizes  the  flow  of  information  to  and  from  the  human  operator  and  provides  the 
appropriate  human  control  interactions  to  the  constructive  system  model. 

Task  3:  Conduct  Trade  Studies.  This  task  involves  the  conduct  of  two  trade  studies  to  select  the 
two  most  appropriate  operational  contexts  from  among  various  predefined  military  domains  of 
interest.  The  objective  of  these  trade  studies  is  to  identify  active  acquisition  programs  that  are 
suitable  for  use  in  the  two  different  case  studies  conducted  under  Task  5. 

Task  4:  Real-Time  Operational  Mission  Simulation.  The  objective  of  Task  4  is  to  modify  and 
prepare  a  mission  simulator  to  represent  the  system  and  environment  for  the  domains  of  interest 
selected  in  Task  3.  This  includes  adding  the  needed  data  collection  capabilities  for  computing 
mission-level  measures  of  effectiveness  (MOEs)  down  through  lower-level  human  performance 
measures. 

Task  5:  Conduct  Two  Case  Studies.  Task  5  involves  the  conduct  of  both  constructive  and  virtual 
simulation  tests  for  identical  mission  tasks  within  the  mission  contexts  selected  under  Task  3. 
These  tests  will  allow  comparison  of  constructive  and  virtual  simulation  results  to  demonstrate 
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the  efficacy  of  the  human  performance  modeling  architecture  and  interface.  The  conduct  of 
these  case  studies  represents  the  heart  of  activity  in  the  CART  program. 

Task  6:  Prepare  Testbed  Definition.  The  final  task  to  be  performed  under  Phase  1  of  the  CART 
program  is  to  develop  a  CART  performance  specification  that  defines  an  implementable  testbed. 
The  specification  will  include  guidance  regarding  how  to  integrate  equipment  or  instrumentation 
in  a  simulation,  the  software  architecture  description,  Run-Time  Infrastructure  (RTI) 
implementation,  and  a  reconfigurable  hardware  architecture  methodology. 

To  date,  Task  1,  Crew  System  Requirements  Establishment,  has  been  completed,  as  has  the 
majority  of  Task  2,  Human  Modeling  Architecture.  Results  of  these  activities,  documented  in 
Brett,  Doyal,  Malek,  Martin  &  Hoagland  (2000),  defined  much  of  the  analytic  process  and 
modeling  architecture  applied  in  Case  Study  1.  The  following  sections  of  this  report  focus  on  the 
conduct  of  this  case  study,  discussing  the  selection  of  the  environment,  development  of  the 
testbed,  test  methodology,  and  results  of  the  data  collection  and  analysis  effort. 
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2.0  THE  CASE  STUDY  1  ENVIRONMENT 


The  purpose  of  the  CART  case  studies  is  two-fold:  (1)  demonstrate  the  technology  of  the  CART 
human  performance  modeling  environment,  and  the  approach  to  model  development  and 
integration  with  constructive  simulations,  and  (2)  assess  the  validity  of  the  CART  concept  by 
comparing  mission  performance  data  from  simulation  trials  that  incorporate  a  CART-developed 
HPM  with  data  from  trials  incorporating  a  HTTL  simulator  operating  in  the  same  environment.  If 
the  CART  approach  and  tools  can  be  demonstrated  to  create  and  integrate  HPMs  that 
successfully  represent  human  performance  in  the  case  study  environments,  CART  can  be  offered 
as  an  expanded  set  of  tools  to  the  organization/modeler/analyst  seeking  to  improve 
representations  of  operator  performance  in  constructive  simulation  environments. 


2.1  Case  Study  Selection 

A  trade  study  process  was  used  to  select  a  topic  for  Case  Study  1.  In  the  trade  study  process, 
potential  topics  were  identified  from  ongoing  or  planned  acquisitions  for  new  or  evolving 
weapons  systems.  A  brief  description  of  the  Case  Study  1  environment  selection  process  is 
provided  below.  For  a  more  detailed  description,  see  Appendix  A.  The  candidate  topics  were 
evaluated  in  terms  of  six  major  factors,  described  below,  that  affect  the  utility  of  a  topic  to  the 
CART  program: 

1.  Types  of  human  performance  to  be  modeled.  The  objective  was  to  select  a 
system  in  which  operators  had  a  significant  role  in  system  performance  and  could 
affect  mission  performance  —  a  system  with  operator  behavior  that  would  be 
challenging  to  model  and  that  would  test  the  viability  of  the  CART  concept. 

2.  Availability  of  existing  svstem/environment  models.  Funding  on  the  CART 
program  is  limited.  A  key  objective  was  to  find  a  program  that  had  an  existing 
simulation  environment  that  included  a  constructive  representation  of  the  system 
and  mission  environment  of  interest.  This  would  permit  us  to  maximize 
investment  in  development  of  a  human  performance  model  and  integration  of  that 
model  with  the  constructive  system/environment  simulation. 

3.  Cost/effort  to  modifv/integrate  a  constructive  simulation.  Given  the  funding 
limitation  noted  above,  it  was  important  to  consider  the  expected  cost  of  (1) 
developing  a  human  performance  model,  (2)  making  modifications  to  the 
constructive  simulation  to  accommodate  interaction  with  the  human  performance 
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model,  and  (3)  integrating  and  testing  the  resulting  human  performance  modeling 
testbed  to  be  sure  an  effective  demonstration  could  be  completed  within  case 
study  funding. 

4.  Availability  of  and  cost/effort  to  condition  H1TL  simulators.  Given  the 
requirement  to  compare  human  model  performance  with  actual  human 
performance,  it  was  necessary  to  identify  simulation  environments  that  possess 
both  an  HTTL  data  collection  capability  and  a  constructive  battlespace 
environment  with  which  a  human  performance  model  could  be  integrated.  Also, 
the  cost  of  any  modifications  required  to  HITL  simulation  had  to  be  within  the 
constraints  of  the  budget. 

5.  Availability  of  data  required  to  generate  performance  measures.  The  requirement 
to  compare  performance  of  the  human  model  with  actual  human  performance 
meant  that  data  had  to  be  available  in  both  the  constructive  and  virtual  test 
environments  to  support  calculation  of  a  common  set  of  performance  measures. 

6.  Propram  matnritv/schedule  fit  with  CART.  Case  studies  had  to  be  accomplished 
within  a  given  timeframe  dictated  by  the  CART  contract.  It  was  necessary  to  find 
a  program  within  which  simulation  resources  needed  by  the  CART  program 
would  be  in  place  and  available  during  the  time  allotted  for  Case  Study  1. 

As  described  in  Appendix  A,  a  variety  of  programs  were  considered  as  case  study  candidates. 
Each  was  evaluated  on  the  above  factors.  In  the  end,  one  program  and  facility  clearly  emerged 
as  the  strongest  candidate.  This  was  the  Virtual  Strike  Warfare  Environment  (VS WE)  hosted  in 
the  Aeronautical  Systems  Center’s  Simulation  &  Analysis  Facility  (SIMAF)  at  Wright-Patterson 
Air  Force  Base,  Ohio.  The  VSWE  had  been  developed  to  support  requirements  development  for 
the  Joint  Strike  Fighter  (JSF)  program.  It  consisted  of  a  complex,  mature,  high  fidelity  virtual 
simulation  and  data  collection  environment  for  studying  effectiveness  of  conceptual  JSFs  in  a 
variety  of  air-to-ground  attack  missions.  It  was  already  developed  and  its  assets  were  available 
within  the  timeframe  required  by  CART.  Pilot  behaviors  required  by  the  JSF  and  exercised  in 
the  VSWE  were  sufficiently  varied  and  complex  to  pose  a  significant  human  performance 
modeling  challenge.  Also,  it  was  determined  that  the  architecture  of  the  flight  simulator  used  in 
the  VSWE  was  suitable  for  integrating  a  human  performance  model.  This  permitted  the  creation 
of  a  constructive  simulation  environment  that  was  very  similar  to  the  virtual  simulation 
environment.  Consequently,  differences  observed  between  the  performance  of  the  human  model 
and  actual  humans  could  not  be  attributed  to  significant  differences  between  the  virtual  and 
constructive  testbeds. 
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Beyond  the  availability  of  simulation  facilities,  the  SIMAF  VSWE  offered  existing  test  sets 
consisting  of  the  following: 

•  Simulation  software  builds  that  represented  a  specific  set  of  JSF  capabilities 

•  Well-defined  scenarios  that  exercised  a  broad  range  of  pilot  behavior  in  the 
context  of  complex  missions 

•  An  extensive  set  of  measures  of  effectiveness  and  measures  of  performance 
that  the  JSF  program  had  defined  to  evaluate  system  effectiveness 

•  All  the  data  collection  and  reduction  capabilities  required  to  compute  the 
performance  measures 

By  reusing  these  test  sets,  the  CART  program  was  able  to  minimize  the  amount  of  development 
required  for  the  constructive  and  virtual  simulations,  and  devote  more  effort  to  the  development 
of  a  human  performance  model  of  significant  complexity. 

2.2  The  VSWE  3B  Mission  Environment  and  Scenario 

A  number  of  VSWE  exercises  have  been  conducted  in  support  of  the  JSF  program.  With  each 
successive  VSWE,  modeling  and  simulation  components  have  continued  to  evolve.  At  the  time 
the  CART  program  first  became  involved  with  the  VSWE  testbed,  the  SIMAF  and  JSF  program 
had  just  recently  completed  VSWE  exercise  “3B.”  Because  of  its  lower  classification  level,  this 
exercise  provided  a  more  accessible  environment  for  the  integration  and  testing  of  the  CART 
system.  As  such,  the  VSWE  3B  environment  served  as  the  specific  baseline  simulation 
environment  for  CART’s  Case  Study  1. 

2.2.1  The  VSWE  3B  Environment 

The  VSWE  3B  environment  consists  of  an  aircraft  simulation  with  a  cockpit  that  allows  HTTL 
control  of  the  aircraft  and  a  mission-level  constructive  model  that  provides  the  mission 
environment  (terrain  features,  threats,  targets,  etc.)  through  which  the  aircraft  simulation  flies 
and  interacts.  The  cockpit  is  called  the  mission  interactive  combat  station  (MICS),  and  the  Joint 
Integrated  Mission  Model  (JIMM)  provides  the  mission  environment.  The  JIMM  is  based  largely 
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upon  the  Synthetic  Warfare  Environment  Generator  (SWEG)  and  includes  capabilities  native  to 
Suppressor,  the  Air  Force’s  long  standing  mission-modeling  environment. 

The  MICS,  shown  in  Figure  1,  is  a  reconfigurable  cockpit  simulator  that  served  as  the  cockpit 
environment  for  Case  Study  1.  The  MICS  was  powered  by  a  SGI  ONYX  2  dual  rack  with  14 
R 10000  CPUs,  two  Infinite  Reality  Pipes  (64  MB  and  128  MB  texture  memory),  890  MB  RAM 
and  18  GB  of  data  storage.  The  cockpit  consisted  of  a  29-inch  monitor  with  touch  screen 
overlays  and  an  F-16  Block  50  stick  and  throttle.  The  single-channel  out-the-window  (OTW) 
imagery  was  projected  on  a  screen  in  front  of  the  cockpit  with  a  head-up  display  (HUD)  overlay. 
Communication  capabilities  were  integrated  between  the  cockpit  and  test  director  area.  Audio 
was  generated  by  the  Fighter  Requirements  Evaluation  Demonstrator  (FRED)  software  and  I/O 
boards  were  installed  on  each  ONYX  2.  The  MICS  uses  the  JIMM  as  the  environment 
generation  system  and  it  was  hosted  on  a  three-processor  (R 10000)  Silicon  Graphics  workstation 
with  256  MB  RAM  and  9  GB  hard  disk. 


Figure  1.  The  SIMAF  Mission  Interactive  Combat  Stations 
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During  the  HITL  trials,  two  cockpits  were  used  to  provide  data  collection  from  two  different 
pilots  flying  different  missions  simultaneously.  A  closer  view  of  one  of  those  cockpits  is  shown 
in  Figure  2. 

The  MICS  cockpit  is  similar  to  the  F-16  in  the  layout  of  flight  controls  and  switches,  but  a  major 
difference  between  the  MICS  cockpit  and  the  F-16  is  the  large  head-down  display  (HDD) 
presented  on  the  29-inch  monitor. 


Figure  2.  A  MICS  Cockpit 


The  primary  software  for  the  MICS  is  comprised  of  the  FRED  software,  the  Camber  radar  toolkit 
for  Synthetic  Aperture  Radar  (SAR),  and  Paradigm’s  Vega  for  OTW  imagery.  In  addition,  an 
integrated  moving  map  capability,  a  real-time  in-flight  route  planner,  and  an  effects-level 
infrared  model  for  targeting  were  implemented. 
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The  FRED  software  was  used  to  simulate  the  generic  JSF  cockpit  environment.  The  FRED 
consists  of  software  components  that  can  be  used  to  simulate  aircraft  systems  using  various 
hardware  components.  For  Case  Study  1,  the  FRED  software  interfaced  with  a  29-inch  monitor 
and  the  Block  50  F-16  stick  and  throttle  controls  in  the  MICS. 

The  FRED  consisted  of  several  software  modules,  each  supporting  a  different  area  of  the 
simulation  environment.  The  environment  models  managed  the  position  orientation  and  velocity 
of  the  ownship  based  on  inputs  from  the  flight  control  system.  These  models  also  managed  the 
movement  and  weapons  deployment  of  all  non-ownship  entities,  and  managed  the  flight  of  all 
weapons  that  were  launched  by  the  ownship  or  by  a  threat.  The  sensor  models  provided  data  to 
simulate  on-board  sensors  including  an  infrared  search  and  track  (IRST),  electronic  support 
measures  (ESM),  identification  friend  or  foe  (IFF),  missile  warning  radar,  and  targeting  sensors. 

The  mission  computer  module  served  as  the  core  avionics  system  for  the  FRED.  It  provided 
navigation  data  and  managed  various  components  of  the  simulation  based  on  pilot  input 
including  steeipoint/route  management,  weapons  stores  management,  tactical  sensor 
management  (e.g.,  sensor  modes  and  fields-of-view),  auto  modes  management,  defensive 
reaction  module  management,  and  sensor  fusion  management.  In  addition,  route  management 
was  augmented  by  a  real-time  mission  planner  (RTMP)  capability.  The  RTMP  offered  the  pilot 
new  routes  when  there  were  changes  in  threat  situation  and/or  deviation  from  the  current  route 
exceeded  a  threshold  distance. 

The  advanced  information  management  system  (AIMS)  component  used  the  Joint  On- 
Board/Off-Board  (JOBOB)  sensor  fusion  software  to  fuse  on-board  and  off-board  sensor  tracks. 
This  system  filtered  out  unwanted  sensor  reports  and  fusion  track  files  to  reduce  fusion 
processing  requirements  and  cockpit  display  clutter. 

The  data  from  the  various  models  within  the  FRED  were  displayed  on  the  HDD  and  the  HUD. 
The  FRED  simulated  the  HDD  and  the  up-front  control  (UFC),  which  were  displayed  on  the  29- 
inch  monitor.  The  HDD  acted  as  a  multi-purpose  display  (MPD)  and  the  monitor  was  equipped 
with  a  touch  sensitive  capability  to  simulate  the  functionality  of  MPD  pushbuttons.  The  HDD 
display  formats  included  a  radar  display,  a  tactical  situation  display  (TSD),  a  targeting  infrared 
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display,  an  aircraft  management  system  display  (AMS),  a  moving  map  display  (MMD),  and 
electronic  flight  instrument  (EFI)  displays.  The  up-front  controls  consisted  of  touch  sensitive 
areas  designed  to  simulate  UFC  pushbuttons,  a  keypad  area,  and  a  data  entry  display.  The  FRED 
also  provided  a  simulated  HUD  that  was  projected  on  a  screen  in  front  of  the  pilot  station. 

The  mission  environment  generator  in  which  the  simulated  aircraft  flew  consisted  of  the  JIMM. 
The  JIMM  is  an  event-stepped,  object-oriented,  general-purpose  conflict  simulation  capable  of 
participating  in  a  network  with  other  simulations,  simulators,  hardware,  and  HITL  systems.  The 
JIMM  was  based  largely  on  the  Synthetic  Warfare  Environment  Generator  (SWEG)  and  included 
capabilities  native  to  Suppressor,  the  Air  Force’s  long  standing  mission-modeling  environment. 

In  Case  Study  1,  the  JIMM  model  ran  the  Generic  Composite  Scenario  (GCS)  as  developed  for 
the  JSF  VS  WE  3B.  The  GCS  represented  the  specified  mission  environment  including  the 
physical  aspects,  physical  influence,  disruption,  and  movement  of  objects  within  the  environment 
(e.g.,  the  target,  other  moving  objects,  roads,  buildings,  terrain  features,  etc.). 

2.2.2  The  Time  Critical  Target  Scenario 

The  VSWE  3B  exercise  examined  JSF  performance  in  several  scenario  contexts,  but  of  particular 
interest  in  CART  Case  Study  1  was  the  attack  of  a  time  critical  target  (TCT).  TCTs  are  high- 
value,  fleeting  targets  such  as  tactical  ballistic  missile  launchers.  The  TCT  attack  mission,  as 
demonstrated  in  Operation  Desert  Storm,  has  proven  quite  challenging  for  strike  fighters  due  to 
its  time-constrained  nature  and  the  pursuit  of  a  relatively  small,  mobile  target  whose  location  is 
uncertain.  Figure  3  depicts  the  general  form  of  the  TCT  scenarios  used  in  VSWE  3B.  The 
scenario  calls  for  the  pilot  of  a  strike  aircraft  to  employ  multiple  sensors  to  acquire  and  attack  the 
mobile  target.  These  sensors  include  real  beam  and  SAR  with  ground  moving  target  indication 
(GMTI),  as  well  as  a  targeting  infrared  (TIR)  system.  During  ingress,  the  pilot  is  required  to 
evade  a  pop-up  threat  that  launches  a  surface-to-air  missile  (SAM)  and  to  subsequently  recapture 
the  ingress  route  and  resume  target  acquisition.  In  addition,  the  pilot  is  required  to  receive  and 
act  upon  an  in-flight  target  intelligence  update  that  provides  a  more  accurate  representation  of  the 
target’s  position.  If  the  target  is  successfully  acquired  prior  to  arrival  at  the  planned  weapon 
release  point,  the  pilot  then  attacks  it.  Otherwise,  the  pilot  is  required  to  perform  a  manual  re- 
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planning  activity  in  which  a  new  route  to  refly  the  target  area  is  developed.  Once  on  this  ‘refly’ 
route,  the  pilot  continues  attempting  to  acquire  and  attack  the  target. 


Original  Planned  Route 
Route  Actually  Flown 


Figure  3.  Illustration  of  Key  TCT  Mission  Components 

It  is  this  scenario  that  drove  the  high-level  requirements  for  developing  the  Case  Study  1  HPM. 
The  HPM  development  effort,  described  in  Section  3.0,  was  focused  on  creating  a  model  that 
could  realistically  represent  pilot  behavior  and  decision  making  associated  with  performing  the 
strike  mission  in  the  above  scenario. 
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3.0  TESTBED  DEVELOPMENT 


Once  the  Case  Study  1  mission  environment  and  scenario  had  been  fully  defined,  the  CART 
testbed  development  effort  was  initiated.  This  effort  began  with  a  mission  decomposition  that 
served  to  break  down  and  organize  the  mission  into  the  various  operator  goals,  functions,  and 
tasks  performed  during  the  course  of  the  specified  mission.  Using  the  CART  software,  the  goals, 
functions,  and  tasks  were  then  implemented  in  a  task  network  model  that  represented  the 
function/task  hierarchy  and  specified  decision  rules  regarding  the  sequences  of  goals,  functions, 
and  tasks  to  be  performed.  To  complete  the  model,  each  task  was  then  characterized  in  terms  of 
task  time  and  accuracy,  release  conditions  and  effects,  and  operator  workload.  In  addition  to  the 
HPM  development  effort,  modifications  were  made  to  the  cockpit/environment  simulation  to 
enable  the  sending  and  receiving  of  data  and  commands  to  and  from  the  constructive  simulation 
environment.  These  activities,  focusing  on  testbed  development,  are  described  at  a  high  level  in 
the  sections  below.  The  actual  details  of  the  HPM  developed  for  Case  Study  1  are  contained  in 
appendices  to  this  report. 

3.1  Mission  Decomposition 

The  effort  to  determine  pilot  goals  and  activities  associated  with  the  specified  scenario  began 
with  a  detailed  mission  decomposition.  It  should  be  pointed  out  that  the  decomposition  focused 
on  and  represented  only  those  goals,  functions,  and  tasks  relevant  to  a  pilot  flying  in  the  VS  WE 
3B  part-mission  simulation  environment.  There  were  a  number  of  constraints  and 
simplifications  in  this  environment  that  would  not  apply  in  a  real-world  flight  task.  For  example, 
there  were  no  system  malfunctions  in  the  particular  VS  WE  scenario,  nor  was  there  a  possibility 
for  an  air-to-air  encounter.  In  addition,  the  simulated  part-task  mission  was  flown  at  30,000  feet 
and  did  not  require  a  takeoff  or  landing.  As  such,  many  pilot  activities  associated  with  these 
events  (e.g,,  visual  scanning  of  the  sky  for  other  aircraft,  visual  scanning  of  the  ground  for 
targets,  checking  of  engine  temperature,  communication  with  air  traffic  control)  were  not 
generally  performed  by  pilots  in  the  simulator.  Because  results  of  the  HPM  trials  were  to  be 
compared  with  that  of  HITL  data  from  the  simulator,  the  decomposition  and  subsequent  HPM 
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also  omitted  these  types  of  activities,  focusing  only  on  those  activities  required  by  the  part-task 
simulation  environment. 

3.1.1  Means-Ends  Decomposition. 

The  mission  decomposition  was  based  loosely  upon  Jens  Rasmussen’s  means-ends  hierarchy, 
and  was  intended  to  identify  and  organize  attributes  of  the  mission  that  were  to  be  subsequently 
incorporated  into  the  HPM  (Rasmussen,  Pejtersen,  &  Goodstein,  1994).  The  organization  of 
these  attributes  is  hierarchical  and  includes  levels  reflecting  the  mission  purpose,  operator  goals, 
functions,  tasks,  and  physical  systems  with  which  the  tasks  are  performed.  To  develop  the  Case 
Study  1  mission  decomposition,  modelers  began  by  conducting  a  series  of  interviews  with  a 
USAF  strike  fighter  pilot.  This  pilot  had  experience  in  the  F-15E  aircraft,  had  served  as  a  subject 
in  the  VSWE  studies  in  support  of  the  JSF  program,  and  had  subsequently  served  as  a  subject 
matter  expert  (SME)  for  the  VSWE  studies.  Thus,  he  had  a  unique  combination  of  insights 
regarding  the  VSWE  mission  environment,  aircraft  cockpit,  and  scenario-specific  tactics.  During 
these  interviews,  the  SME  described  pilot  goals,  functions,  tasks,  procedures  and  decisions 
relevant  to  performing  the  specific  VSWE  3B  mission.  Based  on  information  obtained  in  these 
interviews,  a  baseline  mission  decomposition  was  created.  It  was  subsequently  presented  to  the 
SME  and  to  two  USAF  F-16  pilots  for  review  and  comments.  In  addition,  modelers  consulted 
SMEs  familiar  with  the  cockpit  environment  used  in  the  VSWEs  and  also  reviewed  the 
VSWE  3B  Pilot’s  Manual  to  identify  and  understand  the  specific  equipment  and  procedures  used 
to.  perform  the  specified  pilot  tasks  in  the  simulator. 

The  mission  purpose,  goals,  and  functions  identified  through  the  decomposition  process  are 
illustrated  in  Figure  4.  The  purpose  (to  destroy  the  TCT)  is  supported  by  five  pilot  goals.  The 
Control  Aircraft  / Maintain  Situation  Awareness  (SA)  goal  represents  the  pilot’s  goal  of 
monitoring  his  instruments  to  maintain  awareness  of  the  aircraft  and  mission  status.  The  Control 
Aircraft  / Maintain  SA  goal  can  be  thought  of  as  the  ‘default’  ox  Mission-level  goal  that  is 
typically  active  throughout  the  mission.  The  concept  of  operations  for  the  aircraft  calls  for 
autopilot  flight  in  non-evasion  situations,  and  thus,  this  goal  does  not  include  making  control 
inputs  to  the  aircraft.  The  monitoring  functions  include  listening  to  the  audio  channel,  checking 
the  flight  instruments,  monitoring  mission  progress  and  threats  (i.e.,  the  tactical  situation),  and 
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checking  the  aircraft  system  status.  The  Evade  Threats  goal  represents  the  pilot’s  goal  once  a 
missile  has  been  launched  at  the  aircraft.  It  consists  of  monitoring  the  audio  channel  for 
additional  threat  tones,  evaluating  the  severity  of  the  threat,  selecting  an  evasion  strategy, 
executing  that  strategy  through  a  manual  maneuver  and  use  of  countermeasures,  and  returning 
the  aircraft  to  normal  auto-pilot  flight  upon  completion  of  the  evasive  action.  The  Navigate  goal 
reflects  the  pilot’s  goal  of  changing  the  desired  mission  route.  Within  the  overall  scenario,  this 
goal  includes  functions  of  accepting  a  re-plan  generated  automatically  by  the  mission  planner  or 
requesting  a  manually  generated  mission  re-plan.  Manual  re-plan  functions  include  configuring 
the  planner,  creating  a  plan  to  refly  the  target  area  if  the  target  is  not  detected  and  identified  on 
the  first  pass,  creating  a  plan  to  attack  the  target  once  it  is  identified,  or  planning  to  return  to  base 
(abort)  after  an  attack  or  after  two  failed  attempts  at  detection  and  identification.  The  Acquire 
Target  goal  consists  of  functions  that  support  sensor  employment  for  target  acquisition.  These 
include  updating  target  coordinates,  choosing  a  sensor  and  deciding  where  to  aim  it,  imaging  the 
target,  evaluating  the  resulting  image,  updating  the  shootlist,  and  designating  the  object  if  it  is 
identified  as  the  target.  Finally,  the  Attack  goal  reflects  the  pilot’s  desire  to  maintain  sensor  track 
on  the  target  until  it  is  within  the  weapon  release  envelope,  and  to  subsequently  release  a  weapon 
on  the  target. 

Figures  5  through  9  illustrate  the  lower  levels  of  the  mission  decomposition  for  the  Control 
Aircraft/Maintain  SA,  Evade  Threats,  Navigate,  Acquire  Target,  and  Attack  Target  goals, 
respectively.  Each  of  these  figures  shows  the  pilot  tasks  performed  in  support  of  the  identified 
functions  as  well  as  the  interface  (physical  form),  if  applicable,  with  which  the  pilot  performs  the 
task.  The  numerous  functions  and  tasks  identified  in  the  up-front  mission  decomposition  will  not 
be  discussed  in  detail  here;  however,  each  function  represented  in  the  final  F1PM  is  briefly 
described  and  diagrammed  Appendix  B. 
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Figure  4.  Mission,  Goal,  and  Function  Levels  of  the  Mission  Decomposition 
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Figure  5.  Decomposition  of  the  Control  Aircraft  /  Maintain  SA  Goal 


Figure  6.  Decomposition  of  the  Evade  Threats  Goal 
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Figure  7.  Decomposition  of  the  Navigate  Goal 
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Figure  8.  Decomposition  of  the  Acquire  Target  Goal 
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Figure  9.  Decomposition  of  the  Attack  Target  Goal 


3.1.2  Identifying  Task  Inputs,  Decision  Logic,  and  Pilot  Commands 

Once  pilot  goals,  functions  and  tasks  had  been  identified,  the  process  of  identifying  task 
information  inputs,  decision  logic,  and  command  outputs  began.  Subject  matter  expert 
interviews  and  the  VSWE  3B  Pilot’s  Manual  were  used  to  identify  what  information  is 
required  by  the  pilot  to  perform  the  various  tasks,  how  that  information  is  used  to  make 
decisions,  and  the  results  of  those  decisions  in  terms  of  pilot  interactions  with  the  aircraft 
(i.e.,  pilot  commands).  This  information  would  subsequently  serve  as  input  for  developing 
task  network  model  diagrams,  for  populating  task  effects,  and  for  defining  decision  criteria  in 
the  HPM.  For  each  task  that  was  identified  as  having  an  associated  pilot  decision  or 
command,  the  pilot  information  requirements,  decision  process  and/or  resulting  pilot  action 
were  identified.  An  example  of  this  information  is  shown  in  Table  1.  A  full  listing  of  the 
initial  information,  decision  logic,  and  command  requirements  identified  for  inclusion  in  the 
model  is  presented  in  Appendix  C. 


Table  1.  Example  of  the  Information,  Decision,  and  Pilot  Commands  Identified  for  a 

Given  Task 


F- . '  .  ■  '  ......  1 

|  Goal 

Function  Name 

Task  Name 

Information  in 

Decision 

Command 

Out 

It  current  heading  = 

i 

j  Evade 

Execute  Evasion 
Strategy 

j  Maintain  Evade 
!  Maneuver 

Current  heading, 
desired  evasive 
heading 

desired  evasive 
heading,  then  end  the 
turn.  Else,  continue 

Continue 
Evasive  Turn 

1 

J_ - 1 

to  turn. 

3.2  Human  Performance  Model  Development 

The  completed  mission  decomposition  served  as  the  basis  for  developing  the  human 
performance  model.  The  operator  goals,  functions,  tasks  and  equipment  (physical  forms) 
identified  in  the  decomposition,  as  well  as  the  rules  for  how  they  interact  and  how 
information  is  used  over  the  course  of  a  mission,  defined  the  primary  attributes  that  the 
model  needed  to  possess.  The  sections  below  discuss  the  steps  involved  in  implementing  the 
knowledge  gained  in  this  decomposition  process  to  develop  the  HPM  used  in  Case  Study  1. 
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3.2.1  Diagramming  the  Network. 


As  discussed  earlier,  the  CART  concept  adopts  a  task  network  approach  to  modeling  and 
makes  use  of  a  specially  tailored  task-network  modeling  environment  (i.e.,  the  CART 
software)  to  develop  human  performance  models.  At  the  goal  level,  the  HPM  represents  the 
operator  objectives  and  priorities  that  organize  behavior.  At  the  lower,  function  and  task 
levels,  model  nodes  and  sequences  represent  the  operator’s  procedural  knowledge.  The 
model  development  process  began  with  diagramming  the  goals,  functions,  and  tasks 
identified  in  the  decomposition  to  form  a  series  of  function/task  networks.  Each  goal, 
function,  and  task  was  represented  with  a  single  goal  node,  function  node,  or  task  node, 
respectively,  in  a  network  diagram.  These  nodes  were  structured  hierarchically,  with  goal 
nodes  decomposed  into  function  nodes,  and  function  nodes  decomposed  into  task  nodes.  The 
nodes  were  then  connected  with  arrows  to  specify  the  sequence  of  activities  that  might  occur 
within  a  given  function.  In  many  cases,  multiple  branches  or  pathways  emerged  from  a  node, 
reflecting  potential  branches  of  multiple,  probabilistic,  or  tactical  decisions.  Tactical 
decision  pathways  represented  the  pilot’s  decision  options,  allowing  alternate  paths  through 
the  function/task  networks  based  on  the  ‘state  of  the  world’  and  the  pilot’s  goals  at  the  time  a 
task  node  was  executed.  An  example  of  a  task  network  diagram  can  be  seen  in  Figure  10. 

An  explanation  of  the  task  network  symbology  and  the  full  set  of  task  network  diagrams 
comprising  the  Case  Study  1  HPM  are  contained  in  Appendix  B. 


Figure  10.  Sample  Task  Network  Diagram 


As  with  the  task  decomposition,  the  sources  of  information  for  linking  network  nodes  and 
inserting  tactical  decisions  were  the  pilot  interviews  and  the  VSWE  3B  Pilot’s  Manual. 
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During  this  step,  it  became  necessary  to  add  a  number  of  ‘dummy’  task  nodes  to  the  model. 
These  dummy  nodes,  which  were  subsequently  assigned  zero  time  and  workload,  were  not 
considered  operator  tasks,  but  rather  were  inserted  to  allow  more  modeling  flexibility  such  as 
embedding  tactical  decisions  among  functions  and  the  rejoining  of  multiple  pathways. 

3.2.2  Task  Performance  and  Workload 

The  next  step  in  developing  the  HPM  was  to  populate  each  of  the  task  nodes  in  the  network 

with  data  that  characterized  the  operator  performance  for  that  task.  The  modeling 

environment  allows  this  characterization  across  a  number  of  dimensions  including  a  time 

standard,  accuracy  standard,  accuracy  measure,  mean  time,  standard  deviation  time, 

distribution  type,  mean  accuracy,  standard  deviation  accuracy,  and  workload  values 

encompassing  the  visual,  auditory,  cognitive  and  psychomotor  dimensions.  Because  the 

model  was  to  be  integrated  with  a  constructive  simulation  environment  that,  in  effect, 

provided  the  standard  or  boundary  conditions,  defining  time/accuracy  standards  was  not 

necessary.  For  the  development  of  this  model,  efforts  focused  on  characterizing  only  the 

mean  task  time  and  the  multidimensional  workload  values  for  each  task.  (Task  accuracy  was 

set  to  100%.)  A  listing  of  all  non-dummy  tasks  and  their  associated  time  and  workload 

values  is  presented  in  Appendix  D. 

* 

Values  for  task  times  were  primarily  derived  using  the  micromodels  resident  within  the 
CART  software.  Based  on  the  level  of  detail  at  which  the  task  was  modeled,  the  task  time 
reflected  either  the  time  associated  with  a  discrete  task  represented  in  a  micromodel  (e.g.,  a 
button  press)  or  a  combination  of  tasks  (e.g.,  eye  movement  +  fixation).  In  cases  where  an 
appropriate  micromodel  did  not  exist  in  the  software,  task  times  were  assigned  using  actual 
data  from  observation  of  pilots  performing  the  task  in  the  VS  WE  cockpit.  Further,  a  small 
number  of  task  times  were  entered  as  expressions.  These  values  changed  as  a  function  of 
some  variable  in  the  model.  For  example,  the  time  assigned  to  the  threat  prioritization  task 
varied  as  a  function  of  how  many  threats  were  currently  active. 

Next,  workload  values  were  assigned  to  each  of  the  tasks  in  the  model.  For  each  of  the 
workload  dimensions,  the  CART  software  provides  a  set  of  seven  workload  values  ranging 
from  0.0  to  7.0.  Associated  with  each  workload  value  is  a  verbal  description  of  the  type  of 
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human  activity  that  corresponds  to  the  given  value  (e.g.,  Psychomotor:  2.20  —  Discrete 
Actuation  (button,  toggle,  trigger)).  For  each  task  in  the  model,  a  verbal  description  of  the 
type  of  visual,  auditory,  cognitive,  and  psychomotor  activity  that  most  closely  describes  the 
task  was  identified.  Its  corresponding  workload  value  was  then  assigned  to  the  task. 

3.2.3  Coding  the  Model 

The  next  step  in  the  model  development  process  involved  coding  the  model.  Model  coding, 
described  below,  consisted  of  a  number  of  programming  activities  that  defined  the  internal 
processes  of  the  model. 

3.2.3. 1  Variable  Definition  and  Mapping.  Within  the  model,  two  types  of  variables  are 
defined.  External  variables  are  those  that  are  used  by  both  the  HPM  and  the  constructive 
simulation  environment.  In  this  model,  they  include  such  things  as  entity  positions  and  states 
as  well  as  commands  that  are  sent  from  the  HPM  to  the  constructive  environment  (e.g., 
aircraft  position  data  and  cockpit  control  inputs).  Internal  variables  are  those  used  only 
within  the  HPM  (e.g.,  ‘perceived’  airspeed,  highest  priority  threat).  Using  a  ‘mapper’ 
capability  in  the  CART  software,  variable  names  for  external  variables  defined  in  the  model 
were  then  mapped  to  their  corresponding  variable  name  in  the  constructive  simulation. 
Appendix  E  provides  a  list  of  all  HPM  variables. 

3.2. 3.2  Specifying  Release  Conditions,  Effects,  and  Decision  Rules.  For  each  task,  any 
release  conditions,  beginning  and/or  ending  effects,  and  decision  rules  were  specified. 

Release  conditions  are  expressions  that  must  evaluate  to  ‘true’  before  the  task  can  fire.  For 
example,  a  release  condition  in  the  Evaluate  Re-plan  task  is  that  a  new  re -plan  must  be 
available  for  viewing.  Beginning  and  ending  effects  are  expressions  that  execute  at  a  task’s 
onset  or  conclusion,  respectively.  In  the  HPM,  these  effects  are  often  used  to  set  the  value  of 
a  ‘perceived’  variable  equal  to  that  of  its  truth  data  counterpart,  to  generate  a  ‘command’  to 
be  sent  from  the  HPM  to  the  constructive  simulation,  or  to  call  a  macro  containing  more 
complex  procedures. 

In  addition,  for  any  task  from  which  two  or  more  pathways  emerged,  decision  node  logic  had 
to  be  specified.  Decision  nodes  use  probability  assignments  in  which  the  probability  of 
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executing  each  potential  pathway  is  assigned,  or  tactical  decisions  in  which  expressions  using 
internal  or  external  variables  are  evaluated  to  select  a  particular  pathway.  A  complete  listing 
of  the  actual  code  associated  with  model  release  conditions,  the  effects  (excluding  macros), 
and  the  decision  nodes  is  presented  in  Appendix  F . 

3.2.33  Defining  Macros.  Often,  when  a  relatively  complex  set  of  code  was  required  to 
perform  a  set  of  calculations,  this  code  was  entered  into  a  user-defined  macro.  In  the  HPM, 
such  macros  were  used  for  initializing  variables,  specifying  look  point  data  for  the 
acquisition  process,  and  calculating  the  detection  and  identification  of  objects  in  the  sensor 
field  of  view.  A  brief  description  of  all  the  macros  used  in  Case  Study  1,  as  well  as  their 
actual  code,  is  contained  in  Appendix  G. 

3.23.4  Defining  Priorities,  Action  Rules  and  Goal  Triggers.  Another  coding  activity 
involved  specifying  how  the  various  goal  functions  interact  and  how  they  get  triggered.  Goal 
function  interaction  is  managed,  in  part,  through  a  goal  action  matrix  in  the  software.  This 
matrix  is  used  to  prioritize  the  goal  functions  and  to  specify  the  effect  of  a  newly  triggered 
goal  function  on  the  Mission-level  network  and  on  any  other  goal  function  networks  that  are 
currently  running.  Newly  triggered  goal  functions  can  be  assigned  to  run  simultaneously 
with  other  goal  functions,  to  suspend  any  subordinate  goal  function(s),  or  to  abort  any 
subordinate  goal  function(s).  Table  2  shows  the  goal  action  matrix  developed  for  the  Case 

Study  1  HPM. 


Table  2.  Goal  Action  Matrix  for  Case  Study  1  HPM 


j  Newly  Active  Coal 

1 

Action  Upon  Competing  Coals 

Mission 

Evade 

Navigate 

Acquire 

Attack 

| 

j  Evade 

Interrupt 

Nothing 

Abort 

Abort 

Abort 

Navigate 

Nothing 

Nothing 

Nothing 

Interrupt 

Interrupt 

i 

3 

Acquire 

1  Attack 

Nothing 

Nothing 

Nothing 

Nothing 

Interrupt 

Nothing 

Nothing 

Nothing 

Nothing 

Nothing 

The  first  column  in  the  matrix  shows  the  prioritized  goal  functions,  listed  in  order  from 
highest  to  lowest  priority.  The  SMEs  deemed  Evade  the  highest  priority  goal  function. 

Given  the  scenario,  they  felt  that  pilot/aircraft  preservation  was  more  important  than  target 
prosecution  such  that  all  other  ongoing  activities  should  get  interrupted  or  aborted  once  a 
missile  is  in  the  air  and  evasive  action  is  required.  The  Navigate  goal  function  was  assigned 
the  next  highest  priority  as  it  had  a  direct  impact  on  both  threat  avoidance  and  target 
acquisition.  When  the  Navigate  goal  function  is  triggered,  the  Acquire  and  Attack  goal 
functions  get  interrupted.  The  Acquire  goal  was  given  the  next  highest  priority  since  a 
successful  attack  is  dependent  upon  target  acquisition.  For  the  purposes  of  the  matrix,  a 
newly  triggered  Acquire  goal  is  assigned  to  interrupt  target  Attack.  In  practice,  however,  the 
Attack  function  always  follows  the  Acquire  goal  such  that  they  are  never  active  at  the  same 
time. 

Another  goal  management  activity  involved  specifying  the  trigger  conditions  for  each  goal. 
Trigger  conditions  consist  of  expressions  that,  when  evaluating  to  ‘true’,  trigger  the  onset  of 
a  goal  function.  In  the  model,  they  often  include  statements  that  evaluate  the  physical  state 
of  the  world  (e.g.,  range  to  the  target  area  <=  20NM)  and  the  status  of  the  mission  (target 
found  =  FALSE).  In  addition,  however,  trigger  conditions  must  also  evaluate  current  status 
of  other  higher  priority  goal  functions  {Evade  goal  function  =  not  running).  In  the  version  of 
CART  software  used  for  Case  Study  1  model  development  (CART  version  1.05G),  the  goal 
action  matrix  only  specified  the  impact  of  a  triggered  goal  function  on  other,  currently 
running  goal  functions1.  It  did  not  consider  whether  to  trigger  a  goal  function  based  on  the 
current  status  of  these  other  goals.  Thus,  goal  function  trigger  conditions  used  in  the  Case 
Study  1  HPM  had  to  also  include  arguments  to  evaluate  the  status  of  higher  priority  goal 
functions,  and  to  trigger  the  new  function  only  if  its  action  was  compatible  with  (i.e.,  would 
not  be  suspended  or  interrupted  by)  the  higher  priority  function. 


‘The  CART  software  goal  management  scheme  has  since  been  modified  to  evaluate  the  status  and  action  rules 
of  a  higher  priority  goal  function  prior  to  starting  a  lower  priority  function.  This  change  first  appeared  in 
CART  Software  Version  1.07F. 
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32.3.5  Employing  a  Workload  Management  Scheme.  The  last  step  in  the  model  coding 
process  involved  implementing  a  means  of  managing  pilot  workload  during  the  course  of  the 
mission.  The  goal  management  structure  allows  two  goals  to  be  active  simultaneously.  That 
is,  multiple  functions  and  tasks  that  support  these  goals  can  fire  at  the  same  time.  Humans, 
however,  are  often  limited  in  their  ability  to  perform  multiple  tasks  simultaneously.  For 
example,  they  cannot  examine  two  separate  visual  display  screens  at  the  same  time.  To 
better  reflect  this  limitation  in  the  model,  a  workload  management  scheme  was  implemented 
in  goal  functions  that  had  the  potential  to  run  simultaneously.  This  was  implemented  as  a 
task  management  scheme  that  represented  the  deliberate  distribution  of  visual  resources 
across  tasks  residing  in  multiple  goal  functions. 

Based  on  the  goal  management  matrix  described  above,  the  Mission-level  model  can  run 
simultaneously  with  the  Navigate,  Acquire,  or  Attack  goal  functions.  Thus,  the  Mission-level 
model  needed  a  means  of  time-sharing  with  these  goal  functions.  To  accomplish  this, 
dummy  tasks,  internal  variables,  and  release  conditions  were  used  to  force  the  mission 
network  and  any  competing  goal  function  network  to  ‘take  turns’  when  executing  task  loops. 
For  example,  consider  the  case  in  which  the  Mission-level  network  and  the  Acquire  goal 
function  network  are  running  simultaneously.  At  the  onset  of  a  Mission-level  network  loop, 
an  internal  Mission  workload  variable  is  set  to  ‘true’.  When  the  loop  concludes,  this  variable 
is  set  back  to  “false.”  Meanwhile,  the  release  condition  for  the  first  task  in  the  Acquire  goal 
function  is  that  the  Mission  workload  variable  be  set  to  ‘false’.  Thus,  the  Acquire  goal 
function  can  only  begin  after  the  current  scan  of  the  instruments  is  completed  in  the  Mission- 
level  network.  Once  the  Mission-level  loop  is  completed  and  the  Acquire  loop  is  initiated,  an 
Acquire  workload  variable  is  set  to  ‘true’.  The  Acquire  workload  variable  remains  ‘true’ 
until  the  Acquire  loop  is  completed  (or  until  the  ‘pilot  requests  a  SAR  image,  for  which  he 
must  wait  a  number  of  seconds),  at  which  time  it  is  set  back  to  false  .  The  initial  task  in  the 
Mission-level  network  loop  also  has  as  a  release  condition  that  the  goal  function  workload 
variables,  including  the  Acquire  workload  variable,  must  be  set  to  ‘false’.  Therefore,  as  the 
Acquire  function  completes  a  loop  through  the  network  and  its  workload  variable  is  set  to 
‘false’,  the  Mission-level  network  is  once  again  free  to  begin  another  loop.  This  time-sharing 
process  continues  as  long  as  the  Acquire  function  is  active. 
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3.3  HLA  Interface 


Once  the  HPM  was  developed,  it  was  integrated  with  the  constructive  simulation 
environment  (FRED/JIMM).  Communication  between  the  HPM  and  the  FRED/JIMM 
simulation  occurred  via  the  HLA  RTI.  For  this  effort,  the  Defense  Modeling  and  Simulation 
Office  (DMSO)  HLA  RTI  version  1.3  was  used.  The  HPM  received  data  regarding  system 
and  mission  status  from  the  constructive  system  simulation.  Actions  to  be  implemented  by 
the  system  (e.g.,  maneuver,  target  designation,  weapon  launch)  were  passed  to  the 
constructive  simulation  by  the  task  network  model. 

CART’s  HLA  interface  employs  the  Real-time  Platform  Reference  Federation  Object  Model 
or  RPR  FOM  (Simulation  Interoperability  Standards  Organization,  1999).  While  CART 
modelers  are  able  to  readily  access  entity  state  data  directly  available  in  the  RPR  FOM, 
CART  makes  extensive  use  of  the  FOM’s  Simulation  Management  (SIMAN)  Interactions 
capability.  SIMAN  Interactions  are  used  to  pass  HPM-unique  data  (e.g.,  information 
displayed  on  operator  interfaces  and  inputs  to  operator  controls)  between  the  HPM  and  the 
system  it  is  controlling.  A  graphical  user  interface  enables  a  user  to  define  SIMAN 
Interaction  data  packets  for  a  given  HPM  during  model  development.  The  CART  RTI 
middleware  uses  these  SIMAN  definitions  to  conduct  the  data  exchange  with  the  constructive 
simulation.  Thus,  CART  RTI  middleware  can  remain  constant  while  the  HPM  changes  or  as 
the  CART  tool  is  used  to  develop  new  models  and  integrate  with  new  constructive 
simulations. 

The  CART  team  chose  to  utilize  the  RPR  FOM  over  creating  new  FOMs  or  adopting  an 
Agile  FOM  Framework  (AFF)  for  the  following  reasons: 

(1)  using  the  SIMAN  Interactions,  the  RPR  FOM  can  quickly  and  efficiently  be 
adapted  to  send  much  of  the  non-standard  data  that  will  quite  often  be 
communicated  in  a  CART  Federation, 

(2)  with  the  RPR  FOM,  CART  users  do  not  have  to  possess  the  programming  skills 
required  to  develop  and  maintain  middleware  code, 
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(3)  the  AFF  only  works  if  a  Federate’s  simulation  object  model  (SOM)  is  similar  to 
the  FOM  of  the  Federation  (i.e.,  for  the  mapping  to  work,  the  SOM  must  have  a 
corresponding  concept  within  the  FOM), 

(4)  the  RPR  FOM  is  well  thought  out,  tested,  and  reliable, 

(5)  the  RPR  FOM  is  currently  available  with  no  development  or  maintenance 
expenses,  and 

(6)  the  RPR  FOM  will  most  likely  be  the  FOM  of  choice  for  many  Federates  with 
which  a  CART  user  will  want  to  interact. 

3.4  Constructive  Testbed  Modification/Development 

Model  integration  with  the  constructive  simulation  also  required  modifications  to  the 
FRED/JIMM  component  (see  Figure  1 1).  As  described  in  Section  2.2,  the  FRED  provides 
the  virtual  cockpit  management  software,  controls  the  avionics  and  aero  models,  and 
manages  the  controls  and  displays.  The  JIMM  is  an  event-stepped,  mission-level  modeling 
environment  that  provides  the  terrain,  targets,  and  threats,  as  well  as  all  command,  control, 
computer,  communications,  intelligence,  surveillance,  and  reconnaissance  (C4ISR)  data  and 
events.  The  shared  memory  interface  is  an  area  that  allows  assets  external  to  JIMM  to 
dynamically  interact  with  JIMM.  This  area  typically  resides  in  a  Shared  Common  Random 
Access  Memory  Network  (SCRAMNET)  shared  memory.  In  order  to  interface  the  VSWE 
simulation  with  the  HLA  RTI  and  the  HPM,  a  number  of  modifications  and  extensions  to  the 
VSWE  testbed  were  required. 

The  primary  software  changes  to  the  VSWE  environment  resided  in  the  FRED  component. 
These  changes  included  the  addition  of  a  new  task  and  minor  modifications  to  the  executive 
and  mission  computer.  The  most  significant  modification  to  the  FRED  was  the  addition  of  a 
new  scheduled  task  to  the  FRED  executive  configuration  file.  The  major  focus  of  the  new 
CART  task  was  to  add  a  layer  of  software  that  would  supply  simulation  data  to,  and  interpret 
commands  from,  the  HPM.  The  subsystem,  comprised  of  the  FRED/HPM  interface  software 
and  the  FT -A  RTI  interface  software,  performs  two  major  functions;  these  are  data  exchange 
and  time  management.  Via  calls  to  the  HLA  RTI  interface,  the  FRED/HPM  interface 
software  retrieves  and  packages  simulation  data  to  be  sent  to  the  HPM,  receives  and 
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processes  commands  from  the  HPM,  and  interfaces  with  the  FRED  simulation  utilizing 
several  new  packages  that  allow  the  HPM  to  interact  with  the  aircraft  (i.e.,  actuate  controls  in 
the  cockpit  and  perceive  information  available  on  cockpit  displays).  The  HLA  RTI  interface 
software  provides  the  interface  to  the  HLA  RTI,  sends  and  receives  the  actual  data  packets, 
and  facilitates  time  management. 


Figure  11.  Simple  Depiction  of  the  Integrated  Case  Study  1  Simulation  Environment 

A  minor  modification  to  the  FRED  consisted  of  configuring  the  FRED  Executive  to  run  in 
hold-off  mode  with  JEMM  as  a  slave.  By  using  a  hold-off  mode,  the  CART  task  can  time 
step  the  FRED  when  it  is  allowed  to  run  (e.g.,  when  a  time  advance  grant  is  received), 
allowing  the  constructive  simulation  to  remain  synchronized  with  the  HPM.  In  addition,  the 
Mission  Computer  software  was  modified  slightly  to  include  changes  to  the  Auto-Pilot 
Sequencing  Logic,  the  addition  of  a  Move  Cursor  procedure  that  did  not  latch  to  nearest 
object,  and  added  logic  for  the  Fly-to-Heading  and  Commanded  Airspeed  functions.  Finally, 
two  minor  modifications  were  made  to  JIMM.  First,  the  memory  allocations  for  JIMM  were 
changed  to  increase  JIMM  reliability.  Second,  JIMM  was  changed  to  run  as  a  slave  to  the 
FRED  Executive.  Through  this  mechanism,  the  FRED  tells  JIMM  when  to  freeze  and  when 
to  run. 
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4.0  METHOD 


As  described  earlier,  Case  Study  1  was  conducted  within  the  VSWE  environment  established 
by  the  JSF  program  for  their  air-to-ground  simulation  efforts.  Using  the  VSWE  TCT  mission 
described  earlier,  two  sets  of  data  collection  trials  were  conducted.  The  first  set  consisted  of 
trials  in  which  the  HPM  controlled  the  aircraft  simulation  via  HLA  (the  ‘HPM  condition’). 

In  the  second  data  collection  effort,  eight  trained  pilots  flew  the  same  simulated  missions  as 
those  flown  by  the  HPM.  This  was  called  the  ‘Human-in-the-Loop’  or  ‘HITL  condition’. 
Mission  data  from  the  HPM  and  pilot-commanded  trials  were  then  analyzed  and  compared  to 
determine  whether  results  from  HPM-controlled  simulation  runs  approximated  those  from 
the  pilot-controlled  runs  across  various  mission  performance  dimensions. 

4.1  Case  Study  1  Experimental  Design 

The  experimental  design  for  Case  Study  1  was  a  between-subjects  design  where  operator 
type  (HPM  vs.  HITL)  was  the  primary  factor.  Scenario  was  a  dimension  added  to  increase 
the  variability  of  the  data  within  subjects  and  to  test  the  robustness  of  the  HPM.  The  six 
scenarios  used  in  VSWE  3B  were  re-used  for  this  effort.  Changes  in  the  experimental 
conditions  from  scenario  to  scenario  included  different  planned  routes  to  the  target,  different 
pop-up  threat  locations,  different  target  locations,  and  different  target  update  coordinates.  All 
of  the  scenario  differences  were  minor,  but  they  did  create  differences  in  the  way  each 
scenario  played  out  as  well  as  variability  in  operator  performance.  One  goal  was  to 
determine  whether  the  variability  in  HPM  performance  across  these  six  scenarios  tracked 
with  that  observed  in  the  HITL  trials. 

Each  pilot  in  the  HITL  condition  performed  six  repetitions  of  the  part-mission  TCT  strike 
scenario,  one  trial  for  each  of  six  different  scenario  variations.  The  HPM  also  performed  six 
repetitions  of  each  scenario  variation  to  complete  the  84-cell  matrix.  Table  3  presents  a 
randomized  scheme  for  ordering  subject  exposure  to  scenarios  that  controlled  order  effects. 

In  Table  3,  a  trial  number  filling  each  scenario  cell  denoted  presentation  order. 
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Table  3.  The  Trial  Presentation  Order  For  Constructive  and  Virtual  Subjects 


♦Subject 

Operator 

Type 

Scenario 

1 

Scenario 

2 

Scenario 

3 

Scenario 

4 

Scenario 

5 

Scenario  j 
6 

1 

HITL 

1 

5 

3 

6 

2 

4 

2 

HITL 

5 

3 

4 

2 

6 

1  1 

3 

HITL 

— - — i 

3 

4 

2 

5 

1 

6  J 

4 

HITL 

6 

2 

1 

4 

5 

3  1 

5 

HITL 

2 

1 

6 

3 

4 

5 

6 

HITL 

4 

6 

5 

1 

3 

2 

7 

HITL 

4 

2 

6 

1 

5 

3 

8 

HITL 

3 

5 

6 

2 

4 

1 

9 

HPM 

3 

5 

6 

2 

1 

4 

10 

HPM 

6 

3 

1 

4 

5 

2 

11 

HPM 

1 

2 

6 

3 

4 

5 

12 

HPM 

5 

1 

3 

4 

2 

6 

13 

HPM 

2 

4 

1 

5 

6 

3 

14 

HPM 

3 

6 

5 

i_  * _ a.. 

2 

1 

4 

LIDAyf  triolc 

♦Subjects  1-8  represent  the  8  pilots  in  the  HITL  condition.  ‘Subjects’  9-14  represent  6  blocks  of  HPM  trials. 


each  treated  in  the  analysis  as  a  subject. 


4.2  HITL  Testing 


4.2.1  Participants 

Eight  subjects,  all  with  military,  tactical  fighter  pilot  experience  took  part  in  the  HITL  test 
during  September  and  October  2000.  Seven  of  the  pilots  were  US  AF  reservists  while  one 
was  a  member  of  the  Ohio  Air  National  Guard.  All  of  the  participants  were  qualified  as 
senior  pilots  (at  least  1500  flight  hours)  or  command  pilots  (at  least  3000  flight  hours).  All  of 
the  pilots  had  experience  in  multiple  single-seat  aircraft,  but  their  most  recent  experience  was 
in  either  F-4,  F-16,  or  A- 10  aircraft.  Five  of  the  eight  had  experience  in  the  F-16,  which  was 
critical  given  the  similarity  between  the  simulator  hands-on  throttle  and  stick  (HOT AS)  and 
that  of  the  F-16. 

4.2.2  Apparatus 

The  apparatus  for  the  HITL  testing  consisted  of  the  FRED  cockpit  and  MICS  used  in  the 
VS  WE  environment  described  earlier  in  section  2.2.1. 
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4.2.3  Procedure 


During  the  HITL  phase  of  the  study,  data  collection  was  organized  into  four  sessions  -  with 
two  pilots  in  each  session.  Each  session  included  a  training  phase  followed  by  a  data 
collection  phase.  The  training  phase  consisted  of  an  overview  briefing  followed  by 
familiarization  training  in  the  cockpit.  After  the  training  phase  was  complete,  each  pilot  flew 
six  data  collection  trials  with  the  order  of  the  trials  randomly  determined  to  minimize  order 
effects. 

4.2.3. 1  Training  Phase.  The  training  phase  began  with  an  overview  briefing.  The  overview 
covered  the  purpose  of  the  study,  the  types  of  missions  the  pilots  were  expected  to  conduct, 
and  the  tactics  that  were  to  be  employed  during  various  phases  of  the  missions.  Topics 
addressed  in  this  briefing  included  the  Study  Purpose  and  Objectives,  the  Scenario 
Threat/Target  Data,  Planned  Routes,  and  Target  Acquisition  and  Threat  Avoidance  Tactics. 

After  a  short  question  and  answer  period,  the  pilots  were  immersed  in  the  cockpit  for  the 
familiarization  training.  Familiarization  training  reviewed  interaction  with  displays  and 
controls  in  the  cockpit,  focusing  on  only  those  features  and  capabilities  that  were  needed  for 
the  study  (i.e.,  HOT  AS  usage,  sensor  manipulation,  re-planner  usage).  During  this  segment 
of  training,  both  pilots  were  seated  in  a  cockpit  and  a  trainer  led  each  of  them  through  a 
series  of  exercises  designed  to  demonstrate  the  VSWE  cockpit  switchology.  These  exercises 
included  switching  between  displays  of  interest,  activating  the  different  modes  of  the  radar 
and  infrared  sensors,  observing  moving  target  indications,  slewing  the  cursor  and  designating 
desired  points  of  interest,  interacting  with  the  in-flight  re-planner,  and  entering  data  into  the 
UFC  panel.  The  pilots  were  then  free  to  ask  questions  and  experiment  with  control  and 
display  interaction. 

Once  both  pilots  and  the  trainers  were  satisfied  that  their  knowledge  of  the  HOTAS  and 
HDD  display  interaction  was  adequate,  they  graduated  to  part-task  rehearsal.  During  this 
segment  of  their  training,  each  pilot  rehearsed  sensor  management  activities  and  evasion 
tactics  within  small  segments  of  a  VSWE  3B  unit  interdiction  (UI)  mission.  The  UI  mission 
was  similar  to  the  TCT  mission,  except  that  the  target  was  an  armored  column.  One  of  the 
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elements  in  the  armored  column  was  replaced  with  a  SCUD  to  expose  pilots  to  the  intended 
target  of  the  TCT  scenarios. 

For  sensor  management,  each  pilot  was  instructed  to  employ  the  sensors  to  detect  moving 
targets,  add  moving  target  indications  to  a  shootlist,  and  then  use  the  combination  of  sensors 
to  detect,  identify,  and  designate  a  SCUD  target.  For  evasion,  each  pilot  was  instructed  to  fly 
a  preplanned  route  in  autopilot  until  the  pilot  detected  a  threat  launch.  After  launch  detection 
the  pilot  was  to  determine  the  threat  type  and  implement  a  course  of  action  based  on  the 
threat  type.  If  the  threat  required  evasive  action,  the  pilot  was  to  disengage  the  autopilot, 
release  a  countermeasure  series,  and  turn  to  180  degrees  away  from  the  launch  indication  at 
full  afterburner  until  the  threat  launch  indication  subsided.  These  activities  were  practiced  a 
few  times  until  both  pilots  and  the  trainers  were  satisfied  that  the  pilots  had  reached  an 
appropriate  level  of  proficiency  for  each  part-task. 

Finally,  each  pilot  integrated  the  elements  of  the  part-task  rehearsal  into  a  full  mission 
rehearsal  exercise  using  the  same  UI  mission  that  was  used  during  part-task  rehearsal. 

During  the  full-mission  rehearsal  each  pilot  flew  a  full  mission  similar  to  those  that  he 
experienced  during  data  collection,  forcing  him  to  implement  most  or  all  of  the  part-task 
activities  in  an  integrated  fashion.  Once  the  trainers  and  the  pilots  had  determined  that  each 
pilot  was  proficient  in  the  conduct  of  the  integrated  strike  mission,  the  training  phase  was 
deemed  complete.  Upon  completion  of  the  practice  trials,  each  subject  performed  the  six 
data  collection  trials.  The  order  of  these  trials  was  counter-balanced  across  subjects.  The 
duration  of  each  trial  was  approximately  30  minutes  real  time. 

4.2. 3.2  Data  Collection  Phase.  During  data  collection,  each  pilot/cockpit  had  a  test  director 
and  a  test  conductor.  The  test  director  was  situated  with  the  pilot/cockpit  in  the  room 
housing  both  cockpits  while  each  test  conductor  was  situated  in  the  SIMAF  battle  room.  The 
test  director  was  responsible  for  providing  instruction  to  the  pilot,  answering  pilot  questions, 
and  coordinating  the  selection  of  the  trial  scenario.  The  test  conductor  was  responsible  for 
executing  the  FRED  and  JIMM  software  for  that  pilot/cockpit  and  ensuring  that  the  data 
collection  files  were  archived  appropriately  prior  to  the  next  run.  Communication  between 
each  test  director  and  test  conductor  was  maintained  via  the  intercom  system. 
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On  each  data  collection  trial,  the  test  director  and  test  conductor  for  each  pilot/cockpit 
coordinated  their  efforts  so  that  the  appropriate  scenario  variation  was  flown  for  that 
pilot/cockpit.  Each  trial  began  with  the  aircraft  in  flight  approaching  the  acquisition  legs  of 
the  TCT  mission.  The  mission  computer  was  pre-loaded  with  the  planned  route,  known 
threat  data,  and  the  expected  target  location.  As  the  pilot  followed  the  route  toward  the 
expected  target  location,  he  began  to  search  the  target  area  with  the  onboard  sensors  in  an 
effort  to  acquire  the  TCT.  The  general  strategy  used  by  the  pilots  was  to  employ  the  real 
beam  radar  and  GMTI  to  detect  moving  objects  in  the  target  area.  Once  these  objects  were 
detected,  the  pilots  added  them  to  the  shootlist  for  later  examination  with  the  TIR.  At  a  given 
time  during  ingress,  the  pilot  received  updated  target  coordinates  from  a  simulated  off-board 
source.  He  then  updated  the  mission  computer  to  specify  the  new  estimated  target  location 
and  began  to  focus  imaging  activities  on  the  new  location.  The  pilot  continued  to  attempt 
acquisition  until  he  located  and  identified  the  target  or  aborted  the  mission  due  to  failure  of 
target  identification  after  a  reflight  of  the  expected  target  area.  If  and  when  the  TCT  was 
identified,  the  pilot  attempted  to  designate  it  for  attack,  fly  the  aircraft  to  the  release  point, 
release  the  weapon,  and  egress  from  the  target  area. 

In  addition  to  the  known  threats,  a  pop-up  threat  was  present  along  each  route.  The  surface- 
to-air  threat,  which  was  not  accounted  for  in  the  planned  route,  could  and  did  launch  missiles 
against  the  aircraft,  forcing  the  pilot  to  take  evasive  action.  If  the  pilot  determined  that  the 
launched  missile  posed  a  threat  to  the  aircraft,  the  pilot  performed  evasive  maneuvers  and 
employed  countermeasures  in  an  effort  to  defeat  the  missile. 

Each  trial  also  offered  a  number  of  opportunities  for  in-flight  re-planning.  These  included 
recapturing  the  planned  route  after  threat  evasion  maneuvers,  planning  to  refly  the  target  area 
if  the  target  was  not  found  after  the  first  pass,  planning  a  direct  route  to  the  target  once  it  had 
been  found,  and  aborting  (returning  to  base)  after  attacking  the  target  or  after  two  failed 
attempts  to  find  it.  Re-planning  activities  included  both  acceptance  of  automated  re-plans 
(i.e.,  the  onboard  re-planner  automatically  offered  a  route  which  the  pilot  could  accept  or 
reject),  and  creation  of  manual  re-plans  for  which  the  pilot  manually  inserted  desired 
waypoints  and  then  requested  a  new  plan  that  included  the  specified  points.  Pilots  were 
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instructed  to  accept  all  automated  re-plans  when  offered,  as  the  onboard  re-planner  was 
designed  to  offer  an  optimal  route.  Manual  re-planning  was  necessary  when  a  change  to  the 
destination  was  required.  Manual  re-plans  were  necessary  for  flying  to  the  identified  target 
(if  it  was  too  far  off  the  planned  route)  and  for  reflying  the  target  area.  Pilots  had  to  refly  the 
target  area,  if  after  the  first  pass,  the  target  had  not  been  identified.  In  this  case,  the  pilots 
were  instructed  to  set  up  a  new  acquisition  leg  starting  approximately  30  NM  outside  the 
target  area,  fly  to  the  initial  waypoint  of  that  leg,  turn  inbound  to  the  target  area,  and  resume 
acquisition  activities. 

Each  trial  terminated  once  the  aircraft  successfully  egressed  from  the  target  area  or  the  pilot 
aborted  the  mission  after  two  overflights  of  the  target  area  without  positive  target  acquisition. 
If  the  aircraft  was  intercepted  by  a  ground  threat  during  the  course  of  a  mission,  the  missile 
intercept  was  recorded,  but  the  mission  continued  until  the  pilot  initiated  a  mission  abort  after 
attacking  the  target  or  initiated  an  abort  due  to  a  failure  to  find  the  target  during  each  of  the 
allotted  two  passes  over  the  target  area. 


4.3  HPM  Testing 


4.3.1  Participants 

Constructive  ‘subjects’  were  mapped  to  the  six  trials  conducted  within  each  scenario.  The 
mapping  was  accomplished  by  applying  a  randomized,  without-replacement  trial  order. 

4.3.2  Apparatus 

4.3.2. 1  Hardware.  The  computer  hardware  that  hosted  the  constructive  environment  and  the 
CART  HPM  included  an  Intel  processor-based  PC  with  a  dual-processor  400  MHz 
Pentium  II  CPU  with  256  MB  RAM  running  the  Windows  NT  version  4.0  (Service  Pack  5) 
operating  system,  and  a  Silicon  Graphics  Octane  workstation  with  two  300  MHz  R 12000. 
processors  and  1  GB  RAM  running  the  IRIX  version  6.5  Operating  system.  These  two 
computers  were  networked  through  100-Base-T  networking  cards. 
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4.3.22  CARTHPM  Environment  Software.  The  CART  HPM  software  version  1.05G  was 
hosted  on  the  PC.  The  CART  HPM  software  is  based  on  the  Improved  Performance 
Research  Integration  Tool  (IMPRINT),  a  human-performance  modeling  environment 
developed  by  the  Army  Research  Laboratory’s  Human  Research  and  Engineering  Directorate 
(ARL/HRED).  IMPRINT  is  Government-owned  software  consisting  of  a  set  of  automated 
aids  to  assist  analysts  in  conducting  human  performance  analyses;  it  provided  the  basic 
structure  and  modeling  methodology  for  developing  human  performance  models.  The  CART 
HPM  software  included  a  set  of  extensions  to  IMPRINT  that  enhanced  its  capabilities.  The 
extensions  mainly  consisted  of  an  added  capability  for  inter-model/simulation 
communication  via  the  HLA  RTI  and  the  addition  of  a  goal  orientation  capability  that 
enables  human  performance  modelers  to  represent  the  adaptive,  goal-oriented  nature  of 
human  performance.  The  HPM  communicated  with  other  external  models  via  the  HLA  RTI 
version  1.3  and  took  advantage  of  the  RPR  FOM  Version  0.4/0.5  (DRAFT)  implemented  in 
the  MAK  Technologies’  VR-LINK  3.3  product. 

4.32.3  FRED/JIMM  Software.  As  in  the  HITL  simulation  trials,  JIMM  provided  the 
simulation  environment.  However,  the  cockpit  software  environment  was  scaled  back  to 
incorporate  only  those  components  required  for  use  with  the  HPM.  One  such  component 
was  the  FRED  software,  which  provided  the  basic  representation  of  the  aircraft  and  pilot 
interfaces.  Additional  capabilities  were  added  to  the  FRED,  enabling  it  to  pass  state  data  to 
the  HPM  and  to  reflect  control  inputs  received  from  the  HPM.  A  second  capability  retained 
for  the  HPM  trials  was  the  RTMP  system,  which  was  employed  to  pass  in-flight  re-planning 
data  to  the  HPM.  Within  the  HPM,  some  sensory  processes  that  operated  on  certain  displays 
were  modeled  without  the  actual  use  of  those  displays.  For  example,  the  probabilities  of 
target  detection  and  identification  were  calculated  as  functions  of  sensor  and  sensor  field  of 
view  (FOV)  selected,  range,  the  location  imaged,  and  the  size/type  of  object  within  the  FOV. 
Thus,  the  image  generation  capability  of  the  Camber  radar  and  the  targeting  infrared  system 
were  not  employed. 

4.32.4  Interface  Support  Software.  The  interface  support  software  acted  as  an  intermediary 
between  the  FRED/JIMM  and  the  HPM.  It  obtained  the  current  aircraft  simulation  data. 
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passed  those  data  to  the  HPM,  and  then  updated  the  FRED/JIMM  with  information  as 
commanded  by  the  HPM.  The  data  exchanges  occurred  via  HLA. 

4.3.3  Procedure 

In  the  HPM  conditions,  the  HPM  was  substituted  for  the  pilot  in  the  loop,  processing  data 
from  the  mission  environment  and  commanding  inputs  to  the  simulated  aircraft.  Trials  were 
terminated  according  to  the  same  conditions  mentioned  for  the  HITL  trials.  As  in  the  HITL 
condition,  one  trial  was  conducted  for  each  of  the  six  scenarios.  This  process  was  repeated 
six  times  to  represent  the  six  different  instantiations  of  the  HPM  shown  in  Table  3.  While 
order  effects  were  not  expected  from  the  HPM,  to  maintain  statistical  validity,  the  actual 
trials  were  randomly  selected  from  a  randomized  block  of  trials  without  repetition  and  were 
assigned  subject  numbers  within  scenario. 

4.4  Data  Collection  and  Analysis 

4.4.1  Measures  of  Effectiveness 

The  dependent  variables  for  Case  Study  1  are  listed  in  Table  4.  These  measures  evaluate 
performance  on  the  generalized  functions  that  drove  specification  of  goals.  Originally,  a 
much  broader  set  of  measures  had  been  specified.  Measures  involving  performance  times 
had  been  of  particular  interest.  These  had  included  the  time  taken  to  acquire  and  attack 
targets,  time  to  react  to  threat  launches,  and  time  spent  re-planning.  Problems  with  recording 
time  of  events  within  JIMM  prevented  calculation  of  reaction  to  launch  and  attack  time  data. 
Also,  lack  of  reliable  events  for  collecting  target  detection  and  identification  performance  by 
pilots  in  the  HITL  condition  precluded  the  ability  to  assess  acquisition  times. 

Beyond  problems  with  the  data  collection,  many  of  the  data  generated  by  the  testbed  were 
classified.  While  performance  measures  other  than  those  listed  in  Table  4  could  be 
calculated,  our  ability  to  report  them  is  limited.  The  measures  in  Table  4  were  selected 
because  (1)  they  represent  a  level  of  assessment  that  would  be  of  interest  to  an  acquisition 
program  office  and,  (2)  they  represent  performances  that  were  judged  to  be  relatively 
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independent  and  not  subject  to  obvious  inter-relationships  that  could  cloud  statistical 
interpretation  of  the  results. 

Because  understanding  of  some  of  the  measures  require  an  understanding  of  some  of  the  JSF 
sub-systems,  each  measure  will  be  explained  briefly  here. 

•  #  re-plans  generated  based  on  navigation  error.  Because  most  flight 
control  in  the  JSF  is  performed  by  the  autopilot,  navigation  activity  by  the 
pilot  centers  on  interaction  with  the  auto-router.  When  the  pilot  deviates 
more  than  a  prescribed  distance  (one  mile  for  this  study)  off  the  planned 
route,  the  auto-router  will  compute  an  adjusted  route  and  offer  it  to  the 
pilot  for  acceptance.  In  the  TCT  scenario,  this  situation  would  occur  when 
the  pilot  had  to  evade  a  SAM  launch.  Because  navigation  re-plans  were 
triggered  by  pilot  route  deviation  behavior,  the  number  of  navigation 
re-plans  generated  provided  some  insight  into  the  relative  comparability  of 
route  deviation  between  pilots  and  the  model. 

•  #  re-plans  accepted  based  on  navigation  error.  The  operator  has  the  option 
of  accepting  or  rejecting  a  re-plan  offered  by  the  auto-router.  Because  the 
auto-router  was  considered  to  be  a  key  enabling  technology  in  the  JSF,  the 
operational  concept  was  adopted  that  the  pilot  would  always  accept  a 
re-plan  offered  by  the  auto-router.  This  behavior  was  programmed  into 
the  HPM.  In  HITL  training,  pilots  were  instructed  to  always  accept  plans 
offered  by  the  auto-router.  The  only  exception  to  this  procedure  would  be 
in  those  instances  in  which  a  pilot  was  engaged  in  evasion  maneuvers  and 
a  re-plan  was  offered. 

•  #  re-plans  generated  based  on  threat.  The  auto-router  used  an  electronic 
order  of  battle  (EOB)  database  in  the  mission  computer  to  predict  the 
location  of  threats  and  then  used  data  from  the  onboard  threat  assessment 
system  to  evaluate  the  activity  of  those  threats.  Based  on  activity  of  a 
known  threat  near  the  planned  route,  the  auto-router  might  determine  that 
adjustments  to  the  route  were  necessary  and  offer  a  re-plan  around  the 
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threat.  Generation  of  threat-based  re-plans  was  not  as  directly  driven  by 
pilot  behavior  as  navigation-based  re-plans.  However,  large  differences 
between  pilots  and  the  HPM  in  the  number  of  threat-based  re-plans  could 
suggest  that,  as  scenarios  unfolded,  significantly  different  routing  resulted 
that  led  to  different  exposure  to  threats  -  and  perhaps  even  different 
opportunities  to  acquire  the  target. 

•  #  re-plans  accepted  based  on  threat.  As  with  navigation  re-plans,  the 
operational  concept  was  to  accept  all  threat-based  re-plans  offered  by  the 
auto-router.  This  provided  an  opportunity  to  determine  whether  pilots 
followed  the  operational  concept  as  consistently  as  the  HPM. 

•  #  threat  locks  on  ownshin.  A  threat  ‘lock’  occurred  when  an  enemy  SAM 
was  able  to  acquire  the  JSF  with  an  acquisition  radar.  This  measure 
counted  the  number  of  times  locks  occurred  during  a  trial.  A  given  SAM 
could  lock  on  the  JSF  more  than  once. 

•  #  threat  launches  at  ownship.  This  measure  counted  the  number  of 
missiles  that  were  launched  at  the  JSF  during  a  trial. 

•  %  of  threat  missiles  defeated  bv  ownship.  A  missile  was  defeated  when  it 
did  not  achieve  a  kill  on  the  JSF.  There  were  a  variety  of  reasons  a  missile 
could  be  defeated.  Track  could  be  lost  on  the  JSF.  The  missile  could  run 
out  of  fuel.  The  missile  could  get  close  to  the  JSF  and  detonate,  but 
damage  assessment  algorithms  could  determine  that  the  JSF  was  not 
destroyed.  This  measure  allowed  us  to  assess  the  relative  effectiveness  of 
pilots  and  the  HPM  at  evading  threats. 

•  Probability  of  correct  acquisition.  Ultimately,  target  acquisition  involved 
the  process  of  acquiring  the  Theater  Ballistic  Missile  (TBM)  target  with 
the  TIR  and  designating  it  to  the  weapon  system.  This  measure  assessed 
the  ability  of  pilots  and  the  HPM  to  routinely  and  correctly  acquire  the 
TBM  (as  opposed  to  some  other  object  such  as  a  tank,  or  nothing  at  all). 

•  Range  at  weapon  release.  A  launch  acceptance  region  (LAR)  was 
provided  on  the  TSD  as  graphical  symbol  that  showed  the  effective 
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engagement  envelope  of  the  given  weapon  at  the  current  speed  and 
altitude.  A  simple  indication  of  the  consistency  with  which  LAR  guidance 
was  followed  was  the  range  at  release. 

•  Probability  of  kill  given  attack.  This  was  another  evaluation  of  attack 
effectiveness  that  assessed  whether  a  kill  occurred  given  an  attack  was 
conducted. 


Table  4.  Dependent  Variables  Collected  During  Case  Study  1 


Mission  Function 

Measure  of  Performance 

Navigation 

#  re-plans  generated  based  on  navigation  error' 

#  re-plans  accepted  based  on  navigation  errorf 

#  re-plans  generated  based  on  threat1 

#  re-plans  accepted  based  on  threat1 

Threat  Evasion 

#  threat  locks  on  ownship1 

#  threat  launches  at  ownshipf 

%  of  threat  missiles  defeated  by  ownship1 

Target  Acquisition 

Probability  of  correct  acquisition 

Target  Attack 

Range  at  weapon  release' 

Probability  of  kill  given  attack 

f  Dependent  variables  used  in  the  statistical  analyses  (see  Section  5.1  and  Table  5). 


4.4.2  Data  Sources 

The  data  sources  included  binary  data  files  generated  by  the  FRED  and  JIMM  during  all 
trials,  and  were  supplemented  with  files  generated  by  the  CART  HPM  during  constructive 
runs.  The  data  captured  in  the  FRED  files  were  event-based  data,  such  as  RTMP  interactions 
(i.e.,  re-plan  requests,  re-plan  accepts),  control  and  display  interactions  (e.g.,  display  formats 
selected,  targets  added  to  the  shootlist,  weapon  releases,  etc.)  and  time-based  data  about 
aircraft  position,  attitude,  and  velocity,  which  were  collected  at  a  5  Hz  rate.  Each  event, 
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whether  it  was  time-based  or  event-based  was  time  stamped  with  a  time  value  common  to  all 
data  sources. 

The  JIMM  data  were  captured  in  a  similar  manner,  but  they  tended  to  represent  object-to- 
object  interactions,  such  as  threat  launches,  radar  tracks,  weapon  releases,  and  target  kills. 
The  CART  HPM  data  consisted  of  text  files  generated  by  the  CART  HPM  software  that 
represented  the  initiation,  duration,  and  termination  of  the  HPM  goals,  functions,  and  tasks, 
as  well  as  workload  measures. 

After  each  trial  was  completed,  all  the  files  associated  with  that  trial  were  archived  according 
to  a  predetermined  scheme,  and  were  transferred  to  another  computer  for  reduction  and 
analysis.  All  the  raw  data  were  reduced  and  transformed  into  Microsoft  Excel  97  files  that 
were  subsequently  imported  into  a  Microsoft  Access  97  database  as  tables.  These  tables 
were  then  queried  for  aggregated,  high-level  MOEs  and  lower-level  MOEs  according  to  the 
MOE  hierarchy  and  the  requirements  of  the  statistical  analysis. 
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5.0  RESULTS 


Initial  review  of  the  results  of  pilot  and  HPM  performance  on  the  ten  measures  listed  in 
Table  4  revealed  virtually  no  differences  on  ‘probability  of  correct  acquisition’  and 
‘probability  of  kill  given  attack’.  The  HPM  found  and  correctly  identified  the  target  on  36 
out  of  36  trials  (p  =  1.0)  and  the  pilots  found  and  correctly  identified  the  target  on  47  out  of 
48  trials  (p  =  .98).  Both  pilots  and  the  HPM  killed  the  target  on  every  attack.  Some 
differences  were  observed  on  the  remaining  eight  measures.  Analyses  were  performed  to 
determine  whether  the  differences  were  statistically  significant  when  the  variables  were 
considered  both  individually  and  in  concert.  Demonstrating  whether  a  difference  existed  fell 
to  three  categories  of  measures  as  produced  with  the  aid  of  Statistical  Package  for  the  Social 
Sciences  (SPSS),  release  10.0  (SPSS  for  Windows,  1999):  (1)  Measures  of  central  tendency 
and  dispersion,  (2)  inference,  and  (3)  internal  structure  -  this  last  being  derived  as  a 
byproduct  of  the  other  two.  Measures  of  central  tendency  and  dispersion  (descriptive 
statistics)  included  means,  standard  deviations  and  intercorrelations  among  the  dependent 
measures.  Measures  of  inference  included  repeated  measures  and  multivariate  and  doubly 
multivariate  analyses  of  variance  (MANOVA).  Measures  assessing  whether  internal 
structural  differences  existed  between  the  two  operator  types  included  an  index  developed  by 
the  Department  of  Psychology  at  the  University  of  Akron  in  the  1980’s,  called  the 
congruency  ratio.  As  adapted  for  this  study,  the  congruency  ratio  was  based  on  the 
intercorrelation  matrices.  Also  included  as  an  assessment  of  internal  structure  were  tests  of 
homogeneity  of  variance/covariance  (Levene  and  Box’s  M  tests)  drawn  prior  to  the 
interpretation  of  MANOVA. 

5.1  Descriptive  Statistics:  Measures  of  Central  Tendency  and  Dispersion 

The  eight  dependent  measures  subjected  to  statistical  analysis  were  all  classified.  A  means 
was  needed  to  present  these  data  in  an  unclassified  format  that  hid  the  true  responses  of  the 
operator,  yet  preserved  any  differences  between  operator  types.  A  ‘mean  rank’  procedure 
was  used  that  transformed  each  of  the  eight  dependent  measures  onto  the  same  numeric 
scale.  Table  H-l  of  Appendix  H  shows  an  example  of  how  this  was  done.  This  method  was 
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used  solely  to  provide  means  and  standard  deviations,  but  not  the  correlations.  The 
inferential  results,  that  in  this  report  depict  the  model  effects  at  a  general  level,  are  such  that 
the  actual  data  values  cannot  be  defined.  For  convenience  and  brevity  in  the  following 
discussions,  an  identifying  number  was  assigned  to  each  of  the  dependent  variables  . 
associated  with  a  set  of  ranks.  Table  5  lists  the  dependent  variables  along  with  their 
identifying  numbers. 


Table  5.  Index  of  Dependent  Variable  Identifiers  to  Variable  Names. 


Identifier 

Variable  Name 

DV1 

#  re-plans  generated  based  on  navigation  error 

DV2 

#  re-plans  accepted  based  on  navigation  error 

DV3 

#  re-plans  generated  based  on  threat 

DV4 

#  re-plans  accepted  based  on  threat 

DV5 

#  threat  locks  on  ownship 

DV6 

#  threat  launches  at  ownship 

DV7 

%  of  threat  missiles  defeated  by  ownship 

DV8 

Range  at  weapon  release 

Figure  12  displays  the  overall  average  mean  ranks  and  95%  confidence  intervals  for  the  eight 
dependent  variables  of  interest.  Referring  to  Figure  12,  on  average,  the  HITL  pilots  showed 
poorer  performance  on  several  of  the  dependent  measures;  namely,  higher  #  threat  locks  on 
ownship’,  ‘#  threat  launches  at  ownship’,  and  ‘range  at  weapon  release’  and  lower  ‘%  of 
threat  missiles  defeated  by  ownship’.  The  results  were  not  as  clear-cut  with  the  remaining 
measures.  For  example,  the  pilots  generated  and  accepted  more  navigation  error-related  re¬ 
plans,  while  the  HPM  generated  and  accepted  more  threat-based  re-plans. 

In  regard  to  variability,  the  percentage  of  variation  around  the  mean  —  known  as  the 
coefficient  of  variation  (standard  deviation  divided  by  the  mean  times  100)  -  was  generally 
less  for  the  HPM  than  for  the  HITL.  However,  the  percentages  were  greater  than  expected 
for  the  HPM,  as  one  example,  a  HPM  high  of  24%  compared  to  a  HITL  high  of  31%  for 
‘range  at  weapon  release’.  Even  so,  on  one  measure  the  reverse  was  true;  a  high  of  17%  for 
the  HPM  compared  to  a  HITL  high  of  12%  for  ‘#  threat  launches  at  ownship’.  Moreover,  the 
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results  were  not  consistent  across  all  scenarios.  Perhaps  a  clearer  picture  of  overall 
variability  is  displayed  in  Figure  12.  For  the  first  four  variables  the  length  of  the  confidence 
intervals  around  the  mean  ranks  are  approximately  the  same  for  both  the  HPM  and  the  HITL. 
As  for  the  remaining  variables,  three  -  DV5,  DV7  and  DV8  -  exhibit  somewhat  greater 
variability  for  the  HITL  than  for  the  HPM;  however,  DV6  shows  pronounced  greater 
variability  for  the  HPM.  Also  note  that  the  confidence  intervals  for  the  two  operator 
conditions  overlap  for  all  dependent  variables  except  DV4. 
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Figure  12.  Comparison  of  Operator  Types  on  Overall  Average  Ranks2 


2  Mean  ranks  and  95%  Confidence  Intervals  for  the  eight  dependent  variables.  The  means  are  presented  at  the 
centers,  with  the  upper  and  lower  confidence  bounds  annotated  on  the  whiskers.  Data  for  the  HPM  condition 
are  on  the  left  of  each  panel,  while  data  for  the  HITL  condition  appear  on  the  right. 
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Although  the  scenarios  were  treated  as  equivalent,  some  of  the  scenarios  revealed  results 
opposite  to  the  overall  average.  For  example,  in  scenarios  one  and  two,  the  HPM 
demonstrated  a  greater  number  of  threat  locks  and  launches  on  ownship  (mean  rank  of  43.08 
and  34.17,  respectively,  for  scenario  one,  and  67.92  and  67.58,  respectively,  for  scenario 
two)  than  did  the  pilots  (mean  rank  of  38.06  and  31.81,  respectively,  for  scenario  one,  and 
33.81  and  48,  respectively,  for  scenario  two).  For  scenario  one,  the  pilots  defeated  a  greater 
percentage  of  missiles  than  did  the  HPM.  The  range  at  release  was  further  for  the  HPM  than 
the  pilots  in  scenario  three.  Finally,  the  HPM  generated  and  accepted  more  navigation- 
related  re-plans  than  did  the  pilots  for  both  scenarios  two  and  four.  Further  detail  for  the 
descriptive  statistics  can  be  found  in  Appendix  I,  Table  1-1.  Note  that  measures  of  central 
tendency  and  dispersion  for  ranked  data  typically  involve  medians  and  interquartile 
deviations.  In  this  instance,  however,  because  the  classified  source  data  would  be  described 
by  means  and  standard  deviations,  it  was  deemed  appropriate  to  treat  the  ranks  that  represent 
those  data  similarly. 

5.1.1  Descriptive  Statistics:  Correlations  Among  the  Dependent  Variables 

In  Appendix  J,  Tables  J-l  and  J-2  display  the  correlations  among  the  dependent  measures  for 
the  total  sample  and  by  operator  type,  respectively.  These  correlations  are  based  on  the 
original  classified  data.  Intercorrelations  based  upon  the  total  sample  (N=84  for  the 
correlations  among  DV1  to  DV6,  N=81  for  correlations  with  DV7,  and  N=83  for  correlations 
with  DV8  due  to  missing  data)  provided  the  framework  for  determining  the  multivariate 
models  explained  in  the  next  section.  The  intercorrelations  by  operator  type  also  became 
important  for  a  subsequent  procedure,  the  calculation  of  the  congruency  ratio  as  discussed  in 
a  later  section.  What  is  important  to  note  here  is  that,  in  the  majority  of  cases,  the 
correlations  among  the  dependent  variables  for  HPM  exhibited  similar  strength  and  the  same 
direction  as  the  correlations  for  HITL.  However,  there  were  several  exceptions.  DV3  and 
DV4  correlated  with  all  the  other  variables  exactly  the  same  for  the  HPM  condition,  while 
there  was  some  -  though  not  excessive  -  variation  for  the  pilot  condition;  this  was  also  true 
of  DV1  and  DV2.  There  were  some  instances  of  correlations  of  dissimilar  strength.  For 
example,  the  HPM  number  of  navigation  error-related  re-plans  generated  and  accepted  (DV1 
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and  DV2)  had  highly  positive  and  significant  (p<.01)  correlations  with  the  number  of 
launches  against  ownship  (DV6).  The  respective  correlations  for  the  pilots  were  still  positive 
and  significant,  but  to  a  much  lesser  degree.  Additionally,  the  number  of  locks  and  launches 
(DV5  and  DV6)  inversely  correlated  with  range  at  release  (DV8)  for  the  HPM  to  a  greater 
degree  than  for  the  pilots.  There  were  differences  in  sign  for  several  variables,  i.e.,  one 
correlation  was  positive  while  the  other  was  negative;  however,  with  two  exceptions,  neither 
of  the  correlations  was  significant  (p<05).  Both  number  of  threat-based  re-plans  generated 
and  accepted  (DV3  and  DV4)  showed  a  significant  (pc.Ol),  negative  correlation  with  percent 
missiles  defeated  (DV7)  for  the  HPM,  while  the  respective  correlations  for  the  pilots  were 
nonsignificant  (p>.05)  and  positive.  We  do  not  have  sufficient  data  to  explain  these 
correlation  anomalies.  They  may  have  been  due  to  genuine  internal  differences  between  the 
two  operator  conditions  or  simply  an  artifact  of  the  simulation  environment  or  the 
experiment. 


5.2  Measures  of  Inference 

5.2.1  Multivariate  Analyses 

HPM  task  times  were  held  constant,  so  the  variability  in  the  HPM  condition  across  trials  was 
due  solely  to  the  variability  arising  from  the  stochastic  nature  of  the  simulated  environment.3 
Variability  in  pilot  performance  from  trial  to  trial  contributed  some  additional  variability  in 
the  HITL  condition.  The  multivariate  repeated  measures  analyses  that  were  used  in  making 
inferences  control  for  this  additional  source  of  variability  in  the  HITL  condition.  Performing 
this  analysis  required  examining  beforehand  the  matrix  of  correlations  among  the  dependent 
variables,  finding  those  sets  of  variables  that  correlate  with  each  other  significantly  (p<.05), 
and  forming  models  to  simultaneously  test  the  sets  of  variables  in  what  is  known  as  a  ‘doubly 
multivariate’  analysis. 


3  Although  the  human  performance  task-network  modeling  environment  provides  a  convenient  means  for 
introducing  variability  into  task  times,  this  was  not  done  because  there  were  no  available  data  regarding  task 
variability,  and  any  task  variability  simulated  would  have  been  based  upon  conjecture.  The  confidence  intervals 
presented  in  Figure  12  do  not  suggest  that  overall  variability  was  markedly  different  between  the  HPM  and  the 
HITL  conditions. 
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The  following  example  uses  five  dependent  variables  to  illustrate  the  concept  -  where  is 
used  to  indicate  a  significant  correlation  (at  p<.05): 

DV1<->DV2  DV2<-*DV3  DV3<->DV4  DV4<->DV5 

DV1<->DV3  DV3^DV5 

DV1<->DV5 

From  these  intercorrelations,  three  models  would  result  -  (1)  DV1,  DV2,  DV3;  (2)  DV1, 
DV3,  DV5;  and  (3)  DV3,  DV4,  DV5.  The  technique  used  to  derive  the  models  in  the 
example  is  relatively  straightforward.  DV1  correlates  significantly  with  all  variables  except 
DV4  -  but  a  four-variable  model  is  not  possible  since  DV2  does  not  correlate  with  DV5; 
thus  DV1  exists  in  two  models.  The  second  model  with  DV1  results  from  the  fact  that  DV3 
correlates  with  DV5,  and  both  correlate  with  DV1.  That  leaves  DV4  unassigned.  If  DV4  did 
not  correlate  with  any  of  the  other  variables,  a  model  with  just  DV4  would  be  appropriate; 
however,  DV4  correlates  with  both  DV3  and  DV5,  resulting  in  the  third  model.  This 
concludes  the  example. 

Examining  the  matrix  of  intercorrelations  for  the  entire  sample  in  the  study  (Table  J-l,  as 
noted  in  the  last  section)  revealed  three  sets  of  variables,  forming  the  following  three  doubly 
multivariate  models: 

Model  1  -  DV4,  DV5,  DV6,  DV8; 

Model  2  -  DV3,  DV4,  DV7; 

Model  3  -  DV1,  DV2,  DV5,  DV6 

Tables  6,  7,  and  8  display  the  results  of  the  omnibus  tests  for  each  of  the  three  models. 
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Table  6.  Multivariate  Tests8  for  Model  1: 
Effect  of  Between  Subjects  Operator  Type 


DV4,  DV5, 
DV6,  DV8 

Value 

F 

Hypothesis 

dfb 

Error 

dfb 

Sig. 

Partial  Eta 
Squared 

Population  Eta 
Squared 

Pillai’s  Trace 

.748 

5.924 

4.000 

8.000 

.016 

.748 

.727 

Wilks’  Lambda 

.252 

5.924 

4.000 

8.000 

.016 

.748 

.727 

2.962 

5.924 

4.000 

8.000 

.016 

.748 

.727 

Roy’s  Largest 

Root 

2.962 

5.924 

4.000 

8.000 

.016 

.748 

.727 

a.  Design:  Intercept+OPERTYPE;  Within  Subjects  Design:  SCENARIO 

b.  The  totals  for  the  degrees  of  freedom  (df)  were  less  than  13  due  to  missing  data  for  Model  1 . 


Table  7.  Multivariate  Tests8  for  Model  2: 
Effect  of  Between  Subjects  Operator  Type 


DV3,  DV4, 
DV7 

Value 

D 

Hypothesis  dfb 

Error 

dfb 

Sig. 

Partial  Eta 
Squared 

Population  Eta 
Squared 

Pillai’s  Trace 

.786 

8.586 

3.000 

7.000 

.010 

.786 

.764 

Wilks’  Lambda 

.214 

8.586 

3.000 

7.000 

.010 

.786 

.764 

Hotelling’s 

Trace 

3.680 

8.586 

3.000 

7.000 

.010 

,786 

.764 

Roy’s  Largest 
Root 

3.680 

8.586 

3.000 

7.000 

.010 

.786 

.764 

a.  Design:  Intercept+OPERTYPE;  Within  Subjects  Design:  SCENARIO 

b.  The  totals  for  the  degrees  of  freedom  (df)  were  less  than  13  due  to  missing  data  for  Model  2. 


Table  8.  Multivariate  Tests8  for  Model  3: 
Effect  of  Between  Subjects  Operator  Type 


DVl,  DV2, 
DV5,  DV6 

Value 

F 

Hypothesis  df 

Error 

df 

Sig. 

Partial  Eta 
Squared 

Population  Eta 
Squared 

Pillai’s  Trace 

.736 

6.260 

4.000 

9.000 

.011 

.736 

.716 

Wilks’  Lambda 

.264 

6.260 

4.000 

9.000 

.011 

.736 

.716 

2.782 

6.260 

4.000 

9.000 

.011 

.736 

.716 

Roy’s  Largest 

Root 

2.782 

6.260 

4.000 

9.000 

.011 

.736 

.716 

a.  Design:  Intercept+OPERTYPE;  Within  Subjects  Design:  SCENARIO 
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For  all  three  models,  all  of  the  multivariate  tests  were  significant  at  alpha  levels  less  than  .05. 
Moreover  the  eta-squared  index  values  above  .7  indicate  a  strong  overall  effect.4  Post  hoc 
tests  were  used  to  determine  to  what  the  differences  found  are  attributable.  Barker  and 
Barker  (1984)  advanced  Hummel  and  Sligo’s  premise,  based  on  Monte  Carlo  simulations, 
that  after  executing  an  omnibus  MANOVA  test  and  finding  significance,  conducting  multiple 
individual  tests  of  the  dependent  variables  will  reveal  where  the  differences  exist  and  protect 
the  experiment-wise  alpha  level  (typically  .05).  Others  have  disagreed,  preferring  instead 
simultaneous  test  procedures,  such  as  the  Bonferroni.  Bray  and  Maxwell  have  indicated  that 
either  category  of  post  hoc  tests  proved  appropriate  to  control  Type  I  error.  In  any  event, 
both  categories  of  tests  ended  with  the  same  results  in  this  study.  Tables  9,  10  and  1 1  display 
the  results  of  the  separate  tests  for  each  of  the  dependent  variables  in  Model  1,  Model  2  and 
Model  3,  respectively. 


Table  9.  Tests  of  Between-Subjects  Effects  for  the  Dependent  Variables  of  Model  1 


Source 

Measure 

df 

F 

Sig. 

Partial 

Eta  Squared 

Population 

Eta  Squared 

Operator 

Type 

DV8 

1 

1.862 

E33 

.145 

~0 

DV6 

1 

9.597 

.010 

.466 

.421 

DV5 

1 

.031 

.864 

.003 

~0 

DV4 

1 

13.812 

.003 

.557 

.520 

Error 

DV8 

11 

DV6 

11 

DV5 

11 

DV4 

1  1  | 

Table  10.  Tests  of  Between-Subjects  Effects  for  the  Dependent  Variables  of  Model  2 


Source 

Measure 

df 

F 

Sig. 

Partial 

Eta  Squared 

Population 

Eta  Squared 

Operator 

Type 

DV3 

1 

4.626 

.060 

.340 

-0 

DV4 

1 

21.154 

.001 

.702 

.672 

DV7 

1 

.556 

.475 

.058 

-0 

Error 

DV3 

9 

DV4 

DV7 

9 

4  Population  estimates  of  the  strength  of  the  effect,  eta  squared,  were  derived  from  a  formula  provided  by  Bray 
and  Maxwell  (1990). 
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Table  11.  Tests  of  Between-Subjects  Effects  for  the  Dependent  Variables  of  Model  3 


Source 

Measure 

df 

F 

Sig. 

Partial 

Eta  Squared 

Population 

Eta  Squared 

Operator 

Type 

DV1 

"7 

.405 

.537 

.033 

~0 

DV2 

i 

.052 

.823 

.004  - 

~0 

DV5 

i 

.009 

.927 

.001 

-0 

DV6 

i 

9.227 

.010 

.435 

.391 

Error 

DV1 

12 

DV2 

12 

DV5 

12 

DV6 

12 

Looking  at  the  three  sets  of  tests,  just  two  of  the  eight  dependent  measures  remained 
significant  (p<.05).  These  were  DV4  (#  re-plans  accepted  based  on  threat)  and  DV6  (#  threat 
launches  at  ownship).  The  strength  of  the  effect  population  estimates  (eta  squared)  for  DV4 
ranged  from  .52  in  Model  1  to  .67  in  Model  2,  and  for  DV6,  from  .39  in  Model  3  to  .42  in 
Model  1.  All  four  cases  showed  moderate  effects,  unlike  the  strong  effects  found  in  the 
omnibus  tests.  Note  that  for  nonsignificant  (p<.05)  F  values,  the  population  eta  squares  were 
treated  in  effect  as  zero. 


5.2.2  Outliers 


SPSS  defines  outliers  as  those  values  greater  than  150  percent  of  the  interquartile  range 
added  to  the  value  at  the  75th  percentile,  or  less  than  150  percent  of  the  interquartile  range 
subtracted  from  the  value  at  the  25th  percentile5.  For  example,  if  the  middle  50  percent  of 
data  values  for  some  dependent  measure  ran  from  60  to  90,  the  interquartile  range  would  be 
30  and  values  above  135  and  below  15  would  be  considered  outliers. 

SPSS  identified  six  percent  of  all  the  data  used  in  the  analysis  as  outliers.  Scenarios  2  and  3 
accounted  for  the  overwhelming  majority  of  outliers  in  the  Case  Study  1.  Doing  a  reanalysis 
with  just  the  four  remaining  scenarios  revealed  that  the  omnibus  tests  for  Model  1,  2  and  3 
were  still  significant  (p<.05),  but  DV6  did  not  remain  significant  (p>.05)  as  Tables  12  and  14 


5  The  interquartile  range  is  the  range  between  the  25th  and  75  percentile  values  (i.e.,  the  middle  50  percent  of 
data  values).  Note  that  SPSS  relies  on  “Tukey’s  Hinges”  to  determine  the  25th  and  75th  percentile  values  that  it 
uses  in  identifying  outliers;  these  are  usually  different  values  than  those  that  might  be  found  with  a  common 
frequency  analysis  program. 


55 


show.  However,  the  strength  of  the  effect  for  DV4  increased  from  .52  to  .74  in  Model  1,  and 
from  .67  to  .81  in  Model  2  -  as  shown  in  Table  12  and  Table  13,  respectively.  Also,  DV3  (# 
re-plans  generated  based  on  threat)  was  now  significant  (p<.05)  in  Model  2,  due  in  part  to  a 
smaller  mean  square  error  arising  from  two  thirds  of  the  DV7  s  missing  data  having  been 
associated  with  the  two  deleted  scenarios.  However,  note  that  DV3’s  partial  eta  squared 
value  increased  twice  over  the  original  value  for  Model  2.  Given  that  eta  squared  is  an 
estimate  based  on  the  sum  of  squares,  the  increase  cannot  simply  be  due  to  a  change  in  the 
error  degrees  of  freedom. 


Table  12.  Tests  of  Between-Subjects  Effects  for  the  Dependent  Variables  of  Model  1 

Without  Scenarios  2  and  3 


Source 

Measure 

df 

F 

Sig. 

Partial 

Eta  Squared 

Population 

Eta  Squared 

Operator 

Type 

DV8 

1 

4.180 

.066 

.275 

~0 

DV6 

1 

1.073 

.323 

.089 

~0 

DV5 

1 

.037 

.851 

.003 

~0 

DV4 

1 

34.132 

.000 

.756 

.736 

Error 

DV8 

11 

DV6 

11 

:  DV5 

11 

DV4 

11 

Table  13.  Tests  of  Between-Subjects  Effects  for  the  Dependent  Variables  of  Model  2 

Without  Scenarios  2  and  3 


Source 

Measure 

df 

F 

Sig. 

Partial 

Eta  Squared 

Population 

Eta  Squared 

Operator 

Type 

DV3 

1 

25.579 

.000 

.699 

.674 

DV4 

1 

52.634 

.000 

.827 

.813 

DV7 

1 

.827 

.383 

.070 

~0 

Error 

DV3 

11 

DV4 

11 

DV7 

11 
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Table  14.  Tests  of  Between-Subjects  Effects  for  the  Dependent  Variables  of  Model  3 

Without  Scenarios  2  and  3 


Source 

Measure 

df 

F 

Sig. 

Partial 

Eta  Squared 

Population 

Eta  Squared 

Operator 

Type 

DV1 

1 

.122 

.732 

.010 

-0 

DV2 

1 

.256 

.622 

.021 

-0 

DV5 

1 

.032 

.861 

.003 

-0 

DV6 

1 

1.924 

.191 

.138 

-0 

Error 

DV1 

12 

DV2 

12 

DV5 

12 

DV6 

12 

5.3  Measures  of  Internal  Structure 

A  measure  developed  in  the  Department  of  Psychology  at  the  University  of  Akron  was 
adapted  for  use  as  a  way  of  arriving  at  a  single,  descriptive  metric  for  assessing  the  overall 
commonality  of  variation  between  the  HPM  and  HITL  conditions.  This  metric,  known  as  a 
congruency  ratio,  affords  a  means  to  quantify  the  overall  relationship  between  the  two 
operator  conditions  by  examining  the  intercorrelations  among  the  eight  dependent  variables 
for  each  condition.  The  pattern  of  correlations  within  the  HPM  and  HITL  conditions  are  in 
effect  themselves  correlated  via  the  congruency  ratio  -  which  can  be  viewed  as  a  coefficient 
of  meta  correlation.  The  congruency  ratio  (CR),  as  used  here,  is  defined  in  Equation  (1)  as: 


CR  = 


1  (a-b.) 


where: 

Ai  corresponds  to  cell  i  of  the  correlation  matrix  of  the  HITL  Dependent  Variables 
Bi  corresponds  to  the  corresponding  cell  i  of  the  correlation  matrix  of  the  HPM  Dependent 
Variables 

i  is  summed  across  all  cells  above  the  principal  diagonal  of  the  correlation  matrix  (i.e., 
summed  across  all  intercorrelations) 

n  =  j  (j-l)/2,  for  j  dependent  variables  (i.e.,  n  is  the  total  number  of  intercorrelations) 
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Using  notional  data,  Table  15  and  Equation  (2)  demonstrate  the  computation  of  the 
congruency  ratio  defined  in  Equation  (1). 


Table  15.  Example  Calculation  of  a  Congruency  Ratio  (CR) 


Group  A 

Group  B 

A  •  B 

A2 

B2 

0.30 

0.40 

0.12 

0.09 

0.16 

-0.20 

0.20 

0.04 

0.50 

-0.30 

-0.15 

0.25 

0.09 

-0.30 

-0.80 

0.24 

0.09 

0.64 

-0.40 

0.20 

-0.08 

0.16 

0.04 

0.70 

0.10 

0.07 

0.49 

0.01 

Sums= 

0.16 

1.12 

0.98 

CR  = 


1=1 


0.16 


±a 


/=] 


i=l 


VU2V098 


=  0.15 


(2) 


As  Table  J-2  of  Appendix  J  shows,  there  existed  28  intercorrelations  among  the  dependent 
measures  for  each  operator  type.  The  resulting  value  of  CR  for  this  study  was  0.78. 

Squaring  the  CR  provides  an  index  called  a  coefficient  of  determination  (Neter  & 

Wasserman,  1974)  which,  in  this  case,  is  the  amount  of  variability  in  the  HITL  condition’s 
structural  pattern  of  correlations  that  was  explained  by  the  HPM  condition’s  structural  pattern 
of  correlations.  In  other  words,  the  HPM  accounted  for  61  percent  of  the  variation  in  the 
pilots’  behavior  in  the  HITL  condition. 


Another  way  of  assessing  structural  differences  between  the  operator  types  is  through 
examination,  of  the  variance/covariance  matrices.  The  Levene  test  of  homogeneity  of 
variance  is  an  SPSS  option  available  for  analysis  of  the  MANOVA  models.  Levene’s  test,  a 
univariate  procedure,  compared  the  variance  exhibited  by  the  model  with  that  exhibited  by 
the  pilots  for  each  scenario  within  each  dependent  variable  separately.  For  example,  the 
HPM  variance  in  scenario  one  for  DV 1  was  compared  to  the  HITL  variance  in  scenario  one 
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for  DV1.  For  those  tests  where  the  index  was  found  to  be  not  significant  (p>.05),  equality  or 
homogeneity  of  variance  was  indicated.  Table  16  lists  the  scenarios  passing  (i.e.,  exhibiting 
nonsignificance)  Levene’s  test. 


Table  16.  Results  of  Levene’s  Test 


Dependent 

Scenarios  Exhibiting  Homogeneity  of 

Variables 

Variance  Between  Operator  Types 

DV1 

2,4,6 

DV2 

'  2,4 

DV3 

-  2,4,6 

DV4 

2,6 

DV5 

4,  5,6 

DV6 

2, 4, 5,  6 

DV7 

1,2, 4,5,6 

DV8 

1,2, 4,  6 

As  can  be  seen  in  Table  16,  a  number  of  scenarios  failed  the  Levene  test  —  as  many  as  four 
(DV2  and  DV4)  and  few  as  one  (DV7).  Note  that  scenario  three  failed  for  all  eight 
dependent  variables.  This  scenario  (and  scenario  two,  but  to  a  lesser  extent)  had  the  greatest 
concentration  of  outliers. 

The  Box  M  test  of  homogeneity  of  covariance  is  another  SPSS  option  available  for  analysis 
of  the  MANOVA  models.  Box  M  includes  a  test  of  variance  as  well  as  extending  to  a  test  of 
covariance  of  pairs  of  scenarios  for  all  dependent  variables  in  a  multivariate  model. 

Although  the  scenarios  were  treated  as  equivalent  (given  the  lack  of  experimental  control 
over  them),  differences  in  the  way  each  pilot  or  model  iteration  responded  to  the  individual 
scenarios  made  calculation  of  the  Box  M  test  problematic  (i.e.,  there  was  a  lack  of 
nonsingular  matrices)  for  the  three  doubly  multivariate  models.  Separate  Box  M  tests  for 
each  dependent  variable  were  possible  only  when  considering  just  the  scenarios  that  passed 
Levene’s  test.  The  results  appear  in  Table  17. 
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Table  17.  Results  of  Box  M  Tests 


Dependent 

Variables 

Tests  of  Homogeneity  of  Variance/Covariance 
Between  Operator  Types 

DV1 

Test  not  possible 

DV2 

Box  M=2.091;  F=0.566,  p>.05 

DV3 

Test  not  possible 

DV4 

Test  not  possible 

DV5 

Box  M=6.291;  F=0.746,  p>.05 

DV6 

Box  M=34.979;  F=2.133,  p<.05 

DV7 

Box  M=27.941;  F=0.888,  p>.05 

DV8 

Box  M=17.627;  F=1.033,  p>.05 

Four  of  the  dependent  variables  passed  the  Box  M  test,  showing  equal  variances  and 
covariances  between  the  model  and  the  pilot.  Of  the  three  variables  implicated  in  operator 
differences  by  at  least  one  previous  analysis,  DV6  failed  and  —  due  to  the  lack  of  nonsingular 
matrices  -  DV3  and  DV4  had  no  Box  M  test  available. 

5.4  Summary  of  Statistical  Results 

Three  categories  of  measures  were  used  to  assess  whether  the  HPM  operated  in  the  same  or 
similar  manner  as  the  HITL.  Although  the  measures  of  central  tendency  and  dispersion 
evinced  some  differences,  HPM  coefficients  of  variation  were  similar  to  those  of  the  HITL 
condition  in  many  cases.  For  the  two  dependent  variables  where  the  inferential  post  hoc 
statistical  tests  (with  all  scenarios  included)  found  significance,  coefficients  of  variation  were 
9%  versus  12%  with  DV4  and  17%  versus  12%  with  DV6,  for  the  HPM  and  HITL 
conditions  respectively.  For  DV3,  significance  was  observed  when  scenarios  two  and  three 
were  deleted;  coefficients  of  variation  were  10%  versus  16%  with  DV3,  for  the  HPM  and 
HITL  conditions  respectively.  Further,  the  measure  of  internal  structure  used  to  assess 
commonality  among  all  the  dependent  measures  revealed  good  consonance  between  the 
model  and  the  pilots,  with  the  model  accounting  for  61%  of  the  variation  in  the  pilots’ 
behavior. 

Interaction  between  operator  type  and  scenarios  was  not  posited  as  part  of  the  hypothesis 
tested,  but  evidence  for  differences  between  scenarios  and  the  way  the  two  groups  operated 
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within  a  particular  scenario  could  not  be  ignored  -  as  evidenced  by  the  tests  of  homogeneity 
of  variance/covariance.  The  three  doubly  multivariate  models  were  tested  on  each  scenario 
separately.  For  Model  1,  the  omnibus  tests  resulted  in  no  significance  (p>.05)  for  any  of  the 
scenarios.  For  Model  2,  significance  (p<.05)  was  found  for  scenarios  three,  four  and  five 
(population  eta-squares  of  .637,  .346  and  .411,  respectively).  For  Model  3,  significance  was 
found  for  scenarios  two,  three  and  four  (population  eta-squares  of  .395,  .474,  and  .527, 
respectively).  Understanding  or  categorizing  the  scenarios  would  be  useful  for  future  case 
studies. 
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6.0  DISCUSSION 


6.1  Assessment  of  Overall  Validity  of  the  Model 

Overall  the  results  of  the  descriptive  and  statistical  analyses  indicate  a  high  degree  of  consistency 
between  the  performance  of  the  HPM  and  that  of  actual  pilots.  This  is  particularly  true  regarding 
measures  of  performance  at  the  goal  level.  In  terms  of  the  Acquire  Target  and  Attack  Target 
goals,  there  was  little  or  no  difference  in  the  probability  that  the  target  would  be  located, 
correctly  identified,  and  destroyed.  Differences  were  observed  in  measures  of  Navigate  and 
Evade  Threats  goal  performance.  These  appear  to  be  driven  by  differences  in  the  way  actual 
pilots  and  the  HPM  used  the  auto-router.  Employment  doctrine  used  in  the  trials  dictated  that  -- 
with  the  exception  of  maneuvering  to  evade  a  launched  missile  —  the  auto-router  was  to  be  used 
exclusively  for  making  route  changes.  The  human  performance  model  followed  this  doctrine 
completely.  Pilots  were  less  consistent. 

Observation  of  the  HITL  trials  and  comments  from  the  pilots  indicate  that  the  inconsistent  use  of 
the  auto-router  by  pilots  can  be  attributed  to  quirks  in  the  performance  of  the  auto-router.  The 
auto-router  was  a  beta  version  of  the  capability.  Sometimes  it  created  routes  that  seemed  to  be 
counterintuitive  to  the  situation.  A  good  example  of  this  are  routes  generated  for  target  attack. 
Targets  were  always  identified  using  the  TIR.  Because  of  the  limited  range  of  the  TIR,  the 
aircraft  was  close  to  the  target  when  the  identification  and  subsequent  designation  was  made. 

Once  the  target  had  been  designated,  pilots  were  to  use  the  auto-router  to  create  an  attack  route  to 
the  target.  Often,  this  route  was  not  a  straight  line  or  short  path  to  the  target.  Indeed  a  relatively 
long  route  might  be  generated  that  spiraled  around  the  target,  eventually  passing  over  it.  In  these 
instances,  some  pilots  would  ignore  the  route  and  drive  directly  to  the  target  and  execute  the 
attack.  Since  the  beta  version  of  the  auto-router  did  not  provide  the  capability  to  update  its 
database  with  a  pop-up  threat’s  location,  pop-up  threats  created  situations  that  could  generate 
unusual  routes.  In  these  cases  (particularly  in  scenario  3),  the  pilot  or  HPM  would  successfully 
evade  a  threat,  but  once  the  evasion  was  complete,  the  auto-router  might  then  offer  a  route  that 
took  the  aircraft  either  back  over  or  close  to  the  threat  so  that  another  evasion  maneuver  was 
required  (this  could  occur  if  the  pop-up  threat  was  very  close  to  the  original  flight  path  and  close 
to  a  ‘must  fly’  point  through  which  the  aircraft  had  to  pass).  In  trials  flown  by  the  HPM,  this 
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process  might  be  repeated  several  times  until,  eventually,  the  aircraft  ‘cleared’  the  threat.  Pilots, 
on  the  other'  hand,  would  immediately  understand  the  problem  and  manually  fly  the  aircraft 
around  the  threat  until  a  route  would  be  generated  that  would  not  overfly  the  threat. 

While  agreement  in  performance  of  the  HPM  and  pilots  was  basically  good  at  the  ‘goal’  level,  it 
would  be  appealing  to  know  the  correlation  of  performance  at  lower  levels.  Unfortunately,  this 
has  proven  difficult.  The  challenge  is  in  obtaining  insight  into  the  lower  level  performance  of 
the  live  pilots.  The  most  interesting  aspects  of  operator  performance  are  those  cognitive  and 
perceptual  activities  that  are  tied  to  mission  success  (e.g.,  target  detection  and  identification, 
shoot  list  prioritization).  These  events  are  not  directly  observable  and  are  difficult,  if  not 
impossible,  to  infer  from  the  overt  actions  that  are  collected.  An  option  is  to  pause  a  HITL  trial 
and  question  a  pilot  regarding  factors  such  as  situation  awareness,  decision-making,  workload, 
current  goals,  etc.  However,  this  can  disrupt  performance.  In  the  case  study,  a  conscious 
decision  was  made  to  not  disrupt  the  pilot  and  live  with  the  loss  of  data.  The  assumption  was 
that  agreement  in  performance  at  the  goal  level  would  be  sufficient  because  this  is  the  level  at 
which  most  decision-makers  would  be  most  concerned. 

The  differences  between  the  performance  of  the  HPM  and  the  actual  pilots  that  were  observed 
raise  some  other  interesting  issues  regarding  model  validity.  At  first  blush,  it  might  seem  that 
HPM  was  deficient  because  it  did  not  predict  situations  in  which  operators  would  not  use  the 
auto-router.  Indeed,  using  data  from  the  HITL  runs,  it  would  be  possible  to  modify  the  HPM  so 
it  behaved  more  consistently  with  actual  pilots.  But  -  is  this  a  more  valid  model?  It  would  seem 
that  the  goal  of  human  performance  modeling  within  the  acquisition  process  is  to  predict  the 
performance  of  well-trained  operators,  consistently  applying  well-defined  employment  doctrine 
and  tactics.  For  stealthy  aircraft  such  as  JSF,  technology  such  as  the  auto-router  can  be  crucial 
for  creating  a  survivable  system.  When  testing  the  system,  the  auto-router  needs  to  be  applied 
consistently  so  that  its  effectiveness  can  be  understood  clearly.  A  real-world  problem,  however, 
is  that  technologies  that  get  tested  early  in  acquisition  are  not  always  mature.  They  have  flaws 
and  quirks  like  the  auto-router  used  in  the  case  study.  Real  operators  are  likely  to  respond  to 
these  flaws  by  using  the  technology  inconsistently  and,  in  the  process,  produce  data  that  clouds 
the  evaluation  of  that  technology.  On  the  other  hand,  human  performance  models  can  be 
developed  that  are  completely  consistent  in  their  use  of  system  capabilities  and  that  produce  data 
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that  support  a  cleaner  evaluation  of  those  technologies.  It  is  noteworthy  that  in  this  case  study, 
the  human  performance  model  demonstrated  better  overall  survival  rates  and  survival-related 
performance  (e.g.,  fewer  launches)  than  the  actual  operators.  These  results  suggest  that  even  the 
flawed  auto-router,  when  used  consistently,  enhanced  aircraft  survival.  It  would  seem 
reasonable  to  conclude  that  the  auto-router  technology  has  merit  and  is  worth  pursuing  as  part  of 
the  acquisition  program. 

6.2  Additional  Benefits  Gained  from  Human  Performance  Modeling 

Beyond  providing  a  realistic  representation  of  human  performance,  experience  from  the  case 
study  suggests  that  the  model  development  process  —  as  well  as  the  model  ultimately  developed 
-  can  provide  the  human  system  team  and  the  broader  system  engineering  team  with  other 
benefits  as  well.  One  of  these  benefits  is  insight  into  effective  tactics  for  system  employment. 
Even  though  it  was  developed  based  on  input  from  subject  matter  experts,  the  initial 
implementation  of  the  strike  fighter  pilot  model  proved  to  be  fairly  ineffective  at  finding  the 
time-critical  target.  The  problem  was  that  initial  efforts  to  detect  the  target  used  SAR  patches 
whose  resolution  was  too  gross  to  yield  a  visually  detectable  return.  The  model  development 
team  took  a  step  back  and  re-thought  the  entire  target  acquisition  process.  An  integrated  target 
acquisition  strategy  was  developed  in  which  the  GMTI  and  medium  and  low-resolution  SAR 
images  were  used  to  for  initial  target  detection  at  distances  beyond  effective  TER.  range.  During 
this  phase,  a  list  of  potential  targets  was  developed.  When  the  aircraft  was  within  range,  the  TIR 
was  applied  to  examine  the  potential  targets  and  find  and  identify  the  target.  As  evidenced  by 
the  performance  of  the  model,  these  tactics  for  target  acquisition  proved  to  be  very  effective. 
Their  effectiveness  was  further  validated  when  pilots  in  the  HITL  trials  were  taught  the  same 
tactics  and,  subsequently,  exhibited  target  acquisition  performance  very  close  to  that  of  the 
model.  The  conclusion  of  the  model  development  team  was  that  human  performance  modeling 
could  provide  a  useful  context  for  developing  tactics  that  most  effectively  employ  a  new  system 
or  technology. 

Another  benefit  of  human  performance  modeling  is  that  it  provides  model  developers  and  users 
with  an  intimate  understanding  of  the  performance  required  of  the  operator.  This  understanding 
can  lead  to  insights  such  as  more  effective  function  allocation  between  operators  or  opportunities 
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for  effectively  applying  automation  or  job  aiding  to  support  the  operator.  In  developing  and 
using  the  JSF  pilot  model,  for  example,  it  quickly  became  clear  that  the  most  difficult,  task¬ 
intensive  portion  of  the  job  was  target  acquisition.  Within  target  acquisition  much  time  was 
spent  performing  the  switch  and  control  manipulation  activities  required  to  operate  the  sensors. 
The  team  realized  quickly  that  employment  of  the  TIR  especially  was  driven  by  the  shoot-list 
and  that  this  manipulation  was  repetitious  and  could  be  automated  easily.  The  basic  concept  of 
this  automation  is  that: 

1.  As  the  pilot  builds  the  shoot-list,  an  intelligent  agent  could  observe  the  location  of 
the  airplane  and  the  location  of  points  on  the  ground  of  interest  and  compute  range  to 
those  points. 

2.  When  the  aircraft  comes  into  TIR  range  of  a  point,  the  agent  could  extend  the  TIR, 
command  it  to  look  at  the  point  of  interest  and  generate  and  save  a  medium  and  high- 
resolution  image  of  the  point. 

3.  The  pilot  could  enter  an  exploitation  mode  on  an  MPD  and  page  through  the 
resulting  imagery  looking  for  the  target,  and  once  found,  designate  it  for  attack. 

It  is  expected  that  this  capability  would  significantly  increase  the  number  of  potential  targets  that 
could  be  examined  in  a  pass  through  the  target  area  because  much  of  the  sensor  manipulation 
activity  would  be  off-loaded  from  the  pilot.  Also,  it  would  be  fairly  easy  to  build  a  working 
prototype  of  the  capability  by  reusing  a  portion  of  the  JSF  pilot  model  Acquire  Target  task 
network.  Indeed,  an  important  insight  gained  in  the  first  case  study  is  that  a  human  performance 
model  can  become  the  basis  for  demonstrating  an  automated  or  aiding  capability. 

6.3  Implications  for  How  M&S  is  Applied 

6.3.1  The  Challenge  of  Obtaining  Constructive  System  Representations. 

A  key  concept  in  CART  is  that  integrating  human  performance  models  with  constructive  system 
and  mission  environments  provides  an  opportunity  to  assess  how  operator  performance  can 
impact  broader  system  and  mission  performance.  Consequently,  a  critical  ingredient  for  CART 
success  is  availability  of  constructive  system  models  that  are  sensitive  to  variation  in  the 
performance  of  a  CART  operator  model.  Our  experience  in  this  Case  Study  suggests  that  such 
constructive  system  models  can  be  difficult  to  obtain  and  that  CART  and  other  HBR  modelers 
might  have  to  seek  alternatives. 
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Early  in  the  case  study,  existing,  available  constructive  models  were  reviewed  in  an  effort  to 
identify  any  that  might  be  able  to  interact  with  a  strike  fighter  pilot  model.  This  review 
determined  that  mission  level  models  such  as  Suppressor  offer  the  ability  to  model  a  complete 
range  of  mission  functions.  However,  the  level  at  which  actions  and  events  within  those 
functions  are  modeled  is  not  sufficiently  detailed.  SAM  engagements  and  associated  outcomes, 
for  example,  are  modeled  using  probability  tables.  The  ability  to  have  the  aircraft  interact 
dynamically  with  the  threat  (e.g.,  maneuver  the  aircraft,  apply  countermeasures)  does  not  exist. 
Thus,  it  would  not  be  possible  to  have  an  operator  model  control  a  Suppressor  model  and  play 
out  the  effects  of  operator  decision-making  regarding  threat  evasion.  On  the  other  hand, 
engagement  level  models  such  as  Radar  Directed  Gun  Simulation  (RADGUNS)  and  Enhanced 
Surface-to-Air  Missile  Simulation  (ESAMS)  play  out  the  aircraft  engagement  by  missiles  and 
guns  in  great  detail.  But,  that  is  all  they  do.  They  do  not  address  target  acquisition,  attack  or 
other  important  functions.  Their  scope  is  too  narrow  for  the  CART  JSF  system-modeling 
requirement.  At  the  end  of  the  review,  the  team  concluded  there  were  no  suitable  constructive 
system  models  among  the  existing  constructive  model  set,  and  an  alternative  approach  was 
required. 

The  alternative  approach  selected  was  to  convert  the  virtual  Mission  Interactive  Combat  Station 
from  the  VSWE  into  a  constructive  simulation.  The  result  was  a  high  fidelity  JSF  constructive 
representation  that  was  sensitive  to  operator  model  performance.  Reuse  of  the  MICS  provided 
tremendous  cost  savings  over  developing  a  JSF  model  from  scratch.  Also,  because  the  operator 
model  did  not  require  detailed  visuals  and  operator  station  graphics,  the  constructive  MICS  could 
be  run  on  lower-cost  computing  platforms.  The  original  virtual  MICS  ran  on  a  fourteen- 
processor  Silicon  Graphics,  Inc.  Onyx.  The  re-hosted  constructive  MICS  runs  on  a  two- 
processor  Silicon  Graphics  Octane.  There  is  a  significant  difference  in  cost  between  these  two 
platforms.  We  expect  that  virtual  simulations  will  provide  an  important  source  of  system 
representation  for  other  human  performance  modelers. 

6.3.2  A  New  Paradigm  for  Linking  Traditional  Constructive  and  Virtual  Simulation. 

Traditional  approaches  to  using  modeling  and  simulation  in  system  acquisition  involve  a  mix  of 
constructive  and  virtual  simulation.  Initially,  a  broad  range  of  alternative  system  concepts  are 
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defined  and  evaluated  using  constructive  simulation.  Alternatives  that  generate  desired  levels  of 
performance  are  selected  for  more  detailed  evaluation  using  virtual  simulation.  Virtual 
simulations  are  developed  which  represent  key  capabilities  of  the  alternatives,  test  plans  are 
developed  for  conducting  an  evaluation,  operators  are  obtained,  and  testing  is  conducted.  The 
alternative(s)  judged  to  be  the  most  cost  effective  following  virtual  testing  are  then  pushed 
forward  into  prototyping  and  engineering  development. 

While  the  above  process  offers  significant  improvement  in  the  acquisition  process,  it  still  has 
some  flaws.  The  constructive  models  used  to  screen  system  alternatives  early  in  the  process 
represent  humans  in  a  very  limited  fashion.  It  is  difficult,  if  not  impossible,  to  systematically 
manipulate  factors  of  interest  to  human  system  designers.  One  cannot,  for  example,  represent 
alternative  function  allocations  between  operators  and/or  machine  to  determine  the  optimal 
allocation  scheme  for  a  given  system  design.  Consequently,  current  constructive  modeling  is  of 
little  use  for  resolving  crew  system  design  issues. 

Virtual  simulation  provides  the  obvious  advantage  of  allowing  potential  operators  to  interact 
with  a  system  concept.  Indeed,  this  is  very  important  because  the  insights  and  information 
gained  can  be  extremely  valuable  for  crew  system  design.  Virtual  simulation,  however,  has  its 
limits  too.  Because  of  the  time  and  expense  required  to  develop  and  modify  virtual  simulations, 
it  often  is  not  possible  to  implement  the  full  range  and  combination  of  capabilities  that  might  be 
of  interest  in  a  program.  Consequently,  a  ‘partial  testing  matrix’  is  implemented  and  only  a 
subset  of  all  possible  alternatives  of  interest  is  marked  for  testing.  The  challenge  here  is  to 
‘guess  right’  on  factors  such  as  what  levels  of  performance  of  a  capability  should  be  tested,  what 
combinations  of  capabilities  are  more  important,  etc. 

Another  limitation  of  virtual  simulation  is  the  human  operators  who  participate  in  testing. 

Testing  often  occurs  over  a  short  period  (e.g.,  days  or  weeks).  For  complex  systems,  this  usually 
is  not  enough  time  for  operators  to  become  proficient  in  system  employment.  It  is  difficult  to  use 
data  from  these  operators  to  predict  levels  of  mission  performance  that  can  be  achieved  by  highly 
trained,  proficient  operators.  For  new  systems,  a  concept  of  employment  might  not  exist.  In  this 
case,  some  portion  of  the  test  event  might  be  devoted  to  letting  operators  ‘play’  with  the  system 
in  order  to  test  different  tactics  and  operational  concepts  and  identify  those  with  merit.  While 
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this  provides  valuable  insight  into  effective  system  employment,  it  also  reduces  the  time 
available  for  testing  system  alternatives.  A  secondary  effect  is  that  operators  evolve  their  own 
tactics  and  procedures.  Some  operators  will  develop  more  effective  tactics  than  others.  This  can 
lead  to  significant  variability  in  performance  across  operators,  which  can,  in  turn,  cloud  the 
assessment  and  comparison  of  system  alternatives.  Finally,  it  can  be  difficult  to  get  operators  to 
cooperate  with  tactics  or  employment  procedures  in  some  cases.  A  good  example  of  this  is  the 
auto-router  used  in  this  case  study.  The  employment  concept  for  the  simulated  JSF  was  that 
pilots  would  always  use  the  auto-router  to  control  the  flight  of  the  aircraft,  except  when  under 
engagement  by  a  SAM.  In  actual  use,  pilots  would  sometimes  ignore  this  directive  and  elect  to 
fly  the  aircraft  themselves.  This  was  driven,  in  part,  by  the  fact  that  the  auto-router  was  a  beta 
version  that  sometimes  produced  a  route  that  was  counterintuitive.  Nevertheless,  pilot  behavior 
was  contrary  to  the  procedure  they  were  given.  If  the  focus  of  the  study  had  been  to  evaluate  the 
auto-router,  the  pilot  behavior  would  have  made  it  difficult  to  obtain  a  clean  assessment. 

The  testbed  developed  for  the  case  study  provides  an  interesting  mix  of  capabilities  that  offer  the 
flexibility  and  economy  of  constructive  simulation  and  fidelity  generally  associated  with  virtual 
simulation.  The  result  is  a  simulation  environment  that  solves  the  problems  described  above  and 
-  we  believe  -  offers  a  new  paradigm  for  integrating  constructive  and  virtual  testing.  First,  a 
CART  human  performance  model  linked  to  what  used  to  be  a  virtual  simulator  provides  a  rich 
constructive  environment  for  exploring  operator  issues  associated  with  system  alternatives  and 
concepts.  This  provides  an  acquisition  program  simulation  team  with  an  opportunity  to  resolve 
these  issues  before  proceeding  to  virtual  simulation.  Also,  because  of  its  high  fidelity  system 
representation,  a  CART  testbed  provides  an  opportunity  to  screen  the  system  alternatives 
identified  through  traditional  constructive  simulation,  subjecting  them  to  more  intense  scrutiny 
than  ordinarily  possible  with  constructive  simulation.  Problems  with  an  alternative  can  be 
identified  before  taking  on  the  time  and  expense  of  implementing  it  in  virtual  simulation. 

Indeed,  a  CART  testbed  can  be  used  to  conduct  testing  on  a  complete  test  matrix  prior  to 
conducting  virtual  simulation.  A  partial  test  matrix  can  then  be  applied  in  virtual  testing  in  an 
effort  to  confirm  results  of  constructive  testing.  Because  the  HPM  can  be  driven  by  data 
produced  by  the  models  that  underlie  the  system  simulation  and  does  not  need  detailed  user 
interfaces,  the  cost  of  modifying  the  constructive  system  simulation  to  represent  different  system 
alternatives  can  be  much  less  than  modifying  a  virtual  simulator.  This  permits  implementation 
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and  testing  of  a  greater  range  of  alternatives.  A  CART  testbed  also  can  be  used  to  resolve 
tactics,  concepts  of  operations,  procedures  and  other  system  employment-related  issues.  This 
was  demonstrated  in  the  earlier  discussion  of  how  CART  was  used  to  derive  more  effective 
tactics  for  sensor  employment  in  JSF.  During  virtual  testing,  these  tactics  can  be  taught  to 
participating  operators.  This  makes  their  performance  more  uniform  and  proficient.  It  also 
reduces  the  amount  of  time  required  for  operators  to  train  and  ‘play’  with  the  system,  leaving 
more  time  for  testing.  Finally,  a  CART  HPM  can  be  developed  that  faithfully  applies  tactics  and 
follows  employment  concepts  providing  a  clear  assessment  of  the  technology  or  capability  of 
interest. 

Not  only  is  CART  a  feed-forward  capability  that  bridges  the  gap  between  traditional  constructive 
and  virtual  simulation,  it  can  become  a  feed-back  mechanism  that  exploits  data  collected  in 
virtual  simulation  to  improve  the  CART  HPM  and,  in  turn,  feed  data  back  to  the  traditional 
constructive  simulations.  Once  the  virtual  simulation  runs  are  complete,  the  data  generated  by 
the  operators  and  the  information  gathered  in  post-mission  debriefs  can  be  used  as  appropriate  to 
refine  performance  of  the  CART  models  to  better  reflect  real  operator  performance.  Also,  data 
generated  by  a  CART  testbed  can  be  used  to  update  traditional  constructive  simulations.  For 
example,  a  CART  testbed  can  be  used  to  fly  vectors  past  different  SAM  sites  that  vary 
parameters  such  as  speed,  altitude,  and  bearing  relative  to  the  site  and  then  implement  evasive 
maneuver  when  the  SAM  launches.  Outcomes  of  the  engagements  can  be  recorded  and  the  data 
can  be  used  to  update  SAM  probability  of  kill  tables  in  models  such  as  Suppressor  so  the  data 
more  accurately  represent  effects  of  a  pilot  on  SAM  survival. 

6.4  Assessment  of  the  CART  HPM  Tool  and  Architecture 

In  general,  the  case  study  team  was  pleased  with  the  IMPRINT-based  task  network-modeling 
tool.  The  graphical  user  interface  made  it  easy  to  specify  tasks  and  develop  networks.  Also,  the 
graphically  based  tool  for  mapping  external  variables  to  SIMAN  interactions  proved  easy  to  use 
and,  ultimately,  will  provide  a  significant  savings  to  CART  users  because  they  will  not  have  to 
re-write  the  CART  middleware  each  time  a  new  HPM  is  built. 

Perhaps  the  most  powerful  feature  of  the  tool  was  the  goal  capability  that  was  added  as  part  of 
the  CART  program.  As  expected,  the  goal  structure  yielded  a  HPM  that  responded  dynamically 
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to  changes  in  the  mission  environment.  It  was  interesting  to  observe  how  variations  on  a  core 
scenario  were  able  to  generate  significantly  different  model  performance  in  terms  of  the  number 
of  times  goals  fired  and  the  duration  that  goals  were  active.  This  confirms  the  expectation  that 
an  advantage  of  being  able  to  connect  human  performance  models  to  system  and  mission 
environment  models  is  that  it  provides  the  human  factors  analyst  and,  indeed,  the  entire  system 
development  team  with  tremendous  insight  into  how  the  mission  environment  and  system  design 
drive  performance  of  the  operator. 

While  the  CART  tool  had  many  positive  aspects,  there  were  some  limitations.  One  of  these  was 
the  programming  language  inside  the  model  development  environment.  One  of  the  realizations 
gained  in  the  case  study  was  that  CART  models  require  much  more  programming  than 
traditional  IMPRINT  models.  Some  of  this  is  driven  by  the  interface  with  the  constructive 
system  representation.  A  large  number  of  variables  that  receive  data  (information)  from  the 
constructive  system  simulation  and  pass  actions  back  to  the  simulation  must  be  defined  and 
managed  within  a  CART  HPM.  Equally  important  is  the  need  to  represent  cognitive,  perceptual 
and  other  processes  that  underlie  task  performance.  In  the  JSF  pilot  model,  for  example, 
extensive  code  was  written  that  represented  the  operator’s  perception  of  targets  on  the  sensor 
displays  and  determined  when  target  detection  and  identification  could  occur.  The  language 
does  not  support  complex  conditional  statements,  but  is  limited  to  simple  ‘If-Then-Else’ 
statements.  Also,  it  does  not  support  nesting  of  conditional  statements.  In  order  to  create  nested 
conditionals,  individual  conditional  instruction  segments  are  embedded  in  macros  and  one  macro 
calls  another  that  calls  another,  etc.  It  is  an  awkward  arrangement  that  can  make  debugging  a 
challenge.  A  powerful  addition  to  future  versions  of  the  CART  HPM  development  environment 
would  be  a  more  extensive,  robust  programming  environment  with  good  debugging  tools. 

A  limitation  of  the  overall  CART  testbed  was  speed  of  execution.  The  integrated 
CART/FRED/JIMM  simulation  ran  three  to  four  times  slower  than  real-time.  Since  completing 
the  case  study,  the  Defense  Modeling  and  Simulation  Office  (DMSO)  has  provided  funding  to 
explore  ways  to  make  the  simulation  run  faster.  The  CART  development  team  has  optimized 
elements  of  the  CART  runtime  environment  and  has  been  able  to  increase  performance  to  about 
1.8  times  real-time.  At  this  point  the  obstacle  is  the  FRED/JIMM  simulation,  which  runs  at  1.3 
times  real-time.  The  remainder  of  the  delay  is  driven  by  overhead  associated  the  HLA  RTI  using 
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time  management  services  and  running  regulated  and  constrained.  The  lesson  learned  here  is 
that  if  real-time  performance  is  desired  in  a  CART  simulation,  all  federates  must  be  capable  of 
achieving  better  than  real-time  performance  to  off-set  delays  imposed  by  the  RTI.  For  CART 
HPMs  themselves,  this  does  not  appear  to  be  a  problem.  These  models  tend  to  run  many  times 
faster  than  real-time. 

Beyond  the  tool  itself,  perhaps  the  most  important  insight  gained  is  the  need  for  efficient  and 
effective  data  collection,  management,  and  analysis  tools.  The  CART  concept  for  data  analysis 
is  to  develop  a  hierarchy  of  performance  measures  and  data  that  can  be  used  to  trace  and  evaluate 
how  low-level  operator  performance  impacts  high  level  functions  and  objectives.  While  the  data 
and  measures  for  the  case  study  could  not  be  discussed  in  detail,  it  is  the  opinion  of  the  CART 
team  that  the  performance-measure  hierarchy  does  provide  an  effective  means  for  explaining 
operator  effects.  The  challenge  is  the  level  of  effort  it  takes  to  generate  the  measures.  The 
testbed  developed  for  the  case  study  generates  massive  amounts  of  data  from  the  JIMM  mission 
environment,  the  MICS  system  representation,  and  the  CART  HPM.  Specialized  data  reduction 
software  had  to  be  developed  to  assimilate  these  data  into  a  database  that  could  be  manipulated 
readily.  Additional  software  was  developed  to  generate  summary  performance  measures  and 
statistics.  Even  more  challenging  is  the  need  to  integrate  the  data  sets  so  an  analyst  can  move 
easily  up  and  down  the  hierarchy  exploring  the  data,  developing  an  understanding  of  how  lower 
level  performance  drives  higher  level  outcomes,  and  developing  an  explanation  of  the  results.  It 
is  the  explanation  of  results  that  can  be  particularly  challenging.  This  process  extends  beyond 
the  manipulation  of  data  output  by  the  simulation.  It  requires  that  detailed  knowledge  of  the 
mission  environment  and  mission  scenario,  characteristics  of  the  system  being  tested,  and  the 
operation  of  the  human  performance  model  be  available  during  data  analysis  and  that  it  is 
possible  to  access  portions  of  these  data  to  be  able  to  answer  questions  as  they  arise. 

Within  the  Case  Study,  the  activity  described  above  was  accomplished  through  custom 
developed  software  or  by  more  manual  processing  using  simple  tools  such  as  spreadsheets  or 
documents.  The  labor  intensive  nature  of  this  method  exhausted  resources  programmed  for  the 
data  analysis  rather  quickly  —  limiting  the  range  of  issues  and  questions  that  could  be  explored. 
The  lesson  learned  is  that  CART  testbeds  generate  a  tremendous  amount  of  data  and  information 
about  a  mission  environment,  but  extracting  that  information  and  exploiting  it  to  the  maximum 
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extent  possible  can  be  difficult.  Attention  needs  to  be  directed  to  developing  an  integrated  set  of 
tools  that  make  the  data  reduction  and  analysis  process  more  efficient. 
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ACRONYMS 


A/G  Air  to  Ground 

AIMS  Advanced  Information  Management  System 

AFF  Agile  FOM  Framework 

AMS  Aircraft  Management  System 

AoA  Analysis  of  Alternatives 

ARL  Army  Research  Laboratory 

CART  Combat  Automation  Requirements  Testbed 

C4ISR  Command,  Control,  Computer,  Communications,  Intelligence,  Surveillance,  and 

Reconnaissance 

df  Degrees  of  Freedom 

DMS  Display  Management  Switch 

DMSO  Defense  Modeling  and  Simulation  Office 

DOD  Department  of  Defense 

DOI  Display  of  Interest 

DTO  Defense  Technology  Objective 

EFI  Electronic  Flight  Instrument 

EOB  Electronic  Order  of  Battle 

ESAMS  Enhanced  Surface-to-Air  Missile  Simulation 

ESM  Electronic  Support  Measures 

FMS  Flight  Management  Switch 

FOM  Federation  Object  Model 

FOV  Field  of  View 

FRED  Fighter  Requirements  Evaluation  Demonstrator 

GCS  Generic  Composite  Scenario 

GMTI  Ground  Moving  Target  Indication 

HDD  Head-Down  Display 

HITL  Human-in-the-Loop 

HLA  High-Level  Architecture 

HOTAS  Hands-On  Throttle  and  Stick 

HPM  Human  Performance  Model 

HRED  Human  Research  and  Engineering  Directorate 

HUD  Head-Up  Display 

IFF  Identification  Friend  or  Foe 

IMPRINT  Improved  Performance  Research  Integration  Tool 

IRST  Infrared  Search  and  Track 

JIMM  Joint  Integrated  Mission  Model 
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JOBOB 

Joint  On-Board/Off-Board 

JSF 

Joint  Strike  Fighter 

LAR 

Launch  Acceptance  Region 

M&S 

Modeling  and  Simulation 

MANOVA 

Multivariate  Analysis  of  V ariance 

MDU 

Multi-Function  Display  Unit 

MICS 

Mission  Interactive  Combat  Station 

MOE 

Measure  of  Effectiveness 

MMD 

Moving  Map  Display 

MPD 

Multi-Purpose  Display 

NM 

Nautical  Miles 

OTW 

Out-the-Window 

RADGUNS 

Radar  Directed  Gun  Simulation 

RPR 

Real-time  Platform  Reference 

RTI 

Run-Time  Infrastructure 

RTMP 

Real-Time  Mission  Planner 

SA 

Situation  Awareness 

SAM 

Surface-to-Air  Missile 

SAR 

Synthetic  Aperture  Radar 

SB  A 

Simulation-Based  Acquisition 

SCRAMNET 

Shared  Common  Random  Access  Memory  Network 

SGI 

Silicon  Graphics  Inc. 

SIMAF 

Simulation  and  Analysis  Facility 

SIMAN 

Simulation  Management 

SME 

Subject  Matter  Expert 

SOM 

Simulation  Object  Model 

SPSS 

Statistical  Package  for  the  Social  Sciences 

SWEG 

Synthetic  Warfare  Environment  Generator 

TBM 

Theater  Ballistic  Missile 

TCT 

Time  Critical  Target 

TIR 

Targeting  Infrared 

TMS 

Target  Management  Switch 

TSD 

Tactical  Situation  Display 

UFC 

Up-Front  Control 

UI 

Unit  Interdiction 

VS  WE 

Virtual  Strike  Warfare  Environment 
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INTRODUCTION 


This  document  describes  the  process  used  to  evaluate  candidate  topic  areas  for  Case  Study  1, 
and  records  the  results  of  that  process.  The  results  of  the  process  include  a  Case  Study  topic 
recommendation  and  some  initial  suggestions  regarding  how  the  integration  might  proceed. 
The  sections  in  this  document  include  the  approach  used  to  evaluate  the  topic  area 
candidates,  the  description  of  the  evaluation  factors,  other  significant  issues  affecting  the 
evaluation  factors,  and  the  results  and  recommendations  from  the  evaluation. 

THE  EVALUATION  APPROACH 

This  process  was  carried  out  in  accordance  with  Task  3  of  the  CART  statement  of  work 
(SOW).  The  SOW  suggested  some  candidate  topic  areas,  and  defined  the  factors  on  which  to 
evaluate  the  various  topic  areas.  The  approach  focused  on  the  Joint  Strike  Fighter  (JSF)  as  a 
primary  candidate  for  the  first  CART  case  study.  However,  the  evaluation  did  not  exclude 
other  potential  topic  areas  of  interest  to  the  Air  Force  Research  Laboratory  (AFRL).  These 
other  topic  areas,  listed  in  alphabetical  order,  included: 

•  •  The  B-  IB  bomber  defensive  systems  upgrade  program  (DSUP) 

•  C4ISR  System  of  Systems  (SoS)6 

•  Control  Station  2010 

•  Information  Operations/Sensors 

•  Uninhabited  Combat  Air  Vehicles  (UCAV) 

The  factors  used  to  evaluate  the  candidate  topic  areas  were: 

•  the  types  of  human  performance  to  be  modeled 

•  the  availability  of  existing  system/environment  models 

•  the  cost/effort  to  create  a  constructive  simulation 


6  Although  the  JSF  program  is  considered  a  weapons  acquisition  program,  the  modeling  and  simulation  effort 
currently  being  undertaken  by  JSF  is  aimed  at  understanding  and  defining  this  weapon  system’s  role  in  a  greater 
system  -  that  of  the  Department  of  Defense’s  (DoD)  Command,  Control,  Communications,  Computing, 
Intelligence,  Surveillance,  and  Reconnaissance  (C4ISR)  system.  Therefore,  this  document  refers  to  the  JSF 
modeling  and  simulation  topic  area  as  the  C4ISR  System  of  Systems  (SoS). 
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•  availability  of  and  cost/effort  to  condition  Human-in-the-Loop  (HITL) 
simulators 

•  availability  of  data  required  to  generate  performance  measures 

•  program  maturity/schedule  fit  with  the  CART  program 

Two  other  issues  that  were  considered  in  the  evaluation,  but  not  specifically  addressed  in  the 
SOW,  were  the  CART  team’s  domain  expertise  and  the  topic  area’s  affiliation  with  the 
simulation-based  acquisition  (SBA)  initiative. 

Each  candidate  topic  area  was  rated  on  each  of  the  evaluation  factors,  and  the  results  of  these 
ratings  were  put  into  a  matrix.  The  rows  in  the  matrix  corresponded  to  the  topic  areas,  while 
the  columns  corresponded  to  the  evaluation  factors.  In  addition,  the  factors  were  weighted  to 
reflect  their  relative  importance  in  the  evaluation.  The  weights  were  established  through 
consideration  of  significant  issues  related  to  the  evaluation  factors  that  are  explained  later. 

The  candidate  topic  area  that  evaluated  highest  was  recommended  for  Case  Study  1.  Finally, 
a  development  strategy  was  suggested  for  integrating  the  CART  human  performance  model 
(HPM)  into  the  selected  constructive  environment. 

THE  EVALUATION  FACTORS 

This  section  lists  all  the  evaluation  factors  and  describes  each  one  in  terms  of  this  evaluation. 
They  are  presented  in  no  particular  order. 

Types  of  human  performance  to  be  modeled 

This  factor  referred  to  aspects  of  human  behavior  present  in  a  system  within  the  topic  area  to 
be  modeled.  The  first  aspect  of  human  behavior  was  that  it  must  have  been  complex  enough 
to  be  of  interest  to  CART.  If  the  human  behavior  within  the  system  were  too  simple  it  would 
not  have  been  an  effective  demonstration  of  CART  capabilities. 

The  second  aspect  of  human  behavior  was  that  it  must  have  been  already  defined  sufficiently 
so  that  there  existed  at  least  one  human  behavior  issue  within  the  system  that  the  target 
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program  was  interested  in  modeling.  The  CART  capability  of  human  performance  modeling 
must  have  been  of  interest  to  the  program. 

Availability  of  existing  system/environment  models 

This  factor  referred  to  the  requirement  for  an  existing  constructive  simulation  testbed  within 
the  target  program.  In  addition,  the  environment  model  (or  mission  model)  needed  to  be  able 
to  generate  mission-level  events  that  could  adequately  exercise  human-like  interaction. 
Finally,  the  existing  models  needed  to  be  accessible  to  the  CART  team. 

Cost/effort  to  create  constructive  simulation 

This  factor  was  intended  to  sensitize  the  evaluation  to  the  cost  and  effort  required  for  creating 
or  modifying  an  existing  constructive  simulation  testbed  of  a  candidate  topic  area.  The  goal 
was  to  reduce  the  risk  to  the  program  as  much  as  possible.  The  more  complete  and 
appropriate  the  existing  constructive  environment  was,  the  less  cost,  effort,  and  risk  to  CART 
Case  Study  1  for  building  and  integrating  those  models  with  CART  software. 

Availability  of  and  cost/effort  to  condition  HITL  simulators 

This  factor  was  intended  to  sensitize  the  evaluation  to  the  cost  and  effort  required  for  creating 
or  modifying  an  existing  HITL  simulation  asset  to  simulate,  as  closely  as  possible,  the 
scenarios  played  out  in  constructive  simulation.  The  goal  was  to  reduce  the  risk  to  the  CART 
program  as  much  as  possible.  The  more  complete  and  appropriate  the  existing  HITL 
simulation  environment,  the  less  cost,  effort,  and  risk  to  CART  Case  Study  1  for  building  and 
testing  those  simulators.  In  addition,  this  factor  was  intended  to  account  for  the  expected 
availability  of  the  target  program’s  intended  HITL  simulation  assets.  While  there  was  no 
expectation  that  the  target  program  was  to  own  its  own  HITL  simulation  assets,  this  factor 
captured  the  scheduling  of  simulation  time  on  the  HITL  assets. 

Availability  of  data  required  to  generate  performance  measures 

In  order  for  the  case  study  to  fully  demonstrate  CART  capabilities,  the  modeling  and 
simulation  environments  had  to  allow  for  adequate  data  collection.  In  addition,  data 
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collection  and  reduction  was  to  have  been  relatively  simple  to  implement  with  few  technical 
or  administrative  challenges.  In  this  case,  an  example  of  a  technical  challenge  may  have 
been  the  need  to  author  a  data  collection  engine  from  scratch.  An  example  of  an 
administrative  challenge  may  have  been  that  the  data  were  too  highly  classified  to  be  released 

to  -  or  used  by  -  the  CART  program. 

Program  maturity/schedule  fit  with  CART 

This  factor  referred  to  the  requirement  that  the  target  program  must  have  been  mature  enough 
to  have  already  established  a  modeling  and  simulation  program,  and  that  modeling  and 
simulation  would  be  conducted  within  the  time  frame  established  by  the  CART  case  study 
schedule.  In  addition,  the  modeling  and  simulation  activities  were  to  have  been  at  the 
appropriate  stage  of  the  program  life  cycle  (i.e.,  the  modeling  and  simulation  efforts  were  to 
be  aimed  at  establishing  requirements  at  some  level). 

Team’s  domain  expertise 

Although  this  factor  was  not  one  specified  by  the  statement  of  work,  it  was  the  intent  of  this 
evaluation  to  reduce  risk  to  the  CART  program  by  choosing  a  case  study  topic  area  in  which 
the  team  had  at  least  some  domain  expertise. 

Affiliation  with  Simulation-Based  Acquisition  (SBA)  initiative 

This  factor,  again  not  one  specified  by  the  SOW,  was  intended  to  establish  affiliation  with 
the  SBA  initiative  in  order  to  better  market  CART  within  the  modeling  and  simulation 
community.  It  was  decided  that  the  target  program  should  be  seeking  to  define  and/or  extend 
SBA  methods  and  tools,  and  that  the  target  program  should  help  advocate  the  use  of  CART 
within  the  SBA  community. 

SIGNIFICANT  CASE  STUDY  ISSUES  RELATED  TO  THE  EVALUATION 

FACTORS 

There  were  two  major  issues  not  included  in  the  evaluation  factors,  but  they  affected  the 
weighting  of  individual  evaluation  factors.  The  first  was  implications  surrounding  the 
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requirement  for  virtual  (HITL)  simulation  needed  to  verify  and  validate  the  constructive 
HPM  simulation.  The  second  was  perceived  benefit  of  CART  to  the  target  program,  and 
how  this  perception  may  impact  the  future  of  CART.  These  two  issues  are  now  addressed 
individually. 

Case  study  requirement  for  virtual  simulation  needed  to  verify  and  validate 

constructive  data 

In  order  to  demonstrate  the  validity  of  a  HPM  provided  by  CART,  each  case  study  requires 
that  virtual  simulation  be  conducted  using  the  same  scenarios  played  by  the  constructive 
environment.  A  technical  challenge  -  given  this  requirement  -  is  the  need  for  controlling 
the  differences  between  constructive  and  virtual  worlds  that  could  affect  mission  outcomes  or 
mission  performance.  For  the  validation  exercise  to  be  the  most  effective,  differences  in 
mission  outcomes  need  to  be  attributable  to,  as  much  as  possible,  the  human  operator  in 
virtual  simulation  and  the  HPM  in  constructive  simulation  —  not  to  differences  in  the 
simulation  environments.  To  accomplish  this  end,  the  differences  between  the  constructive 
and  virtual  worlds  need  to  be  as  few  as  possible  and  as  completely  described  as  possible. 
Therefore,  the  factors  associated  with  the  simulation  environments  were  given  the  most 
weight  in  consideration  of  this  extremely  important  issue. 

Perceived  benefit  of  CART  to  target  program  and  target  program  advocacy 

Another  important  issue  considered  in  assigning  evaluation  factor  weights  was  the  expected 
benefit  of  CART  to  the  target  program  to  maximize  the  case  study’s  demonstration  of  CART 
utility  to  the  acquisition  community.  To  accomplish  this  end,  the  target  program  should  have 
had  a  demonstrated  or  stated  need  for  the  technology  being  offered  by  CART.  In  addition,  it 
was  desirable  to  choose  a  target  program  that  understood  and  supported  CART  program 
goals  and  was  willing  to  act  as  an  advocate  for  these  goals  within  the  modeling  and 
simulation  community. 
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RESULTS  AND  RECOMMENDATIONS 


Table  A-l  shows  the  topic  area  evaluation  matrix.  The  columns  correspond  to  the  evaluation 
factors  while  the  rows  correspond  to  the  candidate  topic  areas.  The  factor  category  weights 
are  shown  with  the  column  titles. 


Table  A-l.  Case  Study  1  Topic  Area  Evaluation  Matrix 


Topic  Area 
(category 
weight) 

Human 

Performance 

Complexity 

(4) 

Current 

Modeling 

Infrastructure 

(5) 

mi 

■ 

MM 

HITL  Sim 
Effort 

_ (3) _ 

Sim  Data 
Availability 
(2) 

gi 

BIEiEiLH 

Measure  of 
Probable 
Success 

C4ISR  SoS 

High 

_ £3) _ 

■■ 

n 

UCAV 

■m 

■HI 

■■ 

HH 

igmn 

1 

mmmi 

Control 
Station  2010 

Undefined 

Undefined 

Undefined 

Undefined 

Undefined 

MM 

m 

B-1  DSUP 

Hjg^H 

Low 

_ in _ 

Poor 

_JU _ 

■ 

hhI 

■ 

m 

Information 
Ops / 
Sensors 

Low 

(i) 

Poor 

(D 

High 

(1) 

High 

(D 

is, 

C 

Poor 

(i) 

Fair 

(2) 

3  -  ftnnri 

Moderate 

(2) 

3  -  Hi  ah 

Poor 

(.41) 

Ratings 


3  -  High 
2  -  Moderate 
1  -  Low _ ____ 


2  -  Fair 
1  -  Poor 


2  -  Moderate 
1  -  High 


2  -  Moderate 
1  -  High 


2  -  Moderate 
1  -  Low  ___ 


-Fair 

-Poor 


2 -Fair 
1  -  Poor 


2  -  Moderate 
1  -  Low 


The  ratings  were  developed  from  discussions  with  topic  area  program  personnel  and  from 
research  into  the  candidate  programs.  The  score  called  Measure  of  Probable  Success  is 
expressed  as  a  fraction  of  the  possible  total  points  calculated  by  multiplying  each  topic  area 
rating  by  its  category  weight  and  summing  that  result  for  each  topic  area.  A  1.00  represented 
100%  probability  of  case  study  success,  where  success  was  considered  as  meeting  all  the 
requirements  for  conducting  a  case  study  under  the  terms  of  the  contract  and  demonstrating 
that  CART  technology  was  beneficial  to  the  modeling  and  simulation  community. 

Topic  Area  Evaluation  Matrix  Results 

The  C4ISR  SoS  scored  very  well  primarily  due  to  the  close  coupling  of  its  constructive  and 
virtual  modeling  and  simulation  environments,  but  the  other  factors  were  also  rated  highly. 
The  UCAV  topic  area  scored  lower  primarily  because  this  program  is  extremely  early  in  its 
life  cycle,  and  because  the  program  currently  has  a  strong  contractor  influence.  UCAV  will 
likely  be  a  top  candidate  during  the  evaluation  of  topic  areas  for  Case  Study  2.  The  Control 
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Station  2010  topic  area,  an  advanced  command  and  control  workstation  concept,  scored 
lowest  primarily  due  to  the  current  lack  of  a  clear  program  definition.  It  is  expected  that  this 
topic  area  will  also  be  revisited  during  the  Case  Study  2  evaluation.  The  B-l  DSUP  topic 
area  evaluated  lower  than  some  of  the  other  topic  areas  primarily  due  to  the  lack  of  a  stated 
need  by  the  B-l  program  for  human  performance  modeling,  especially  in  the  DSUP  program. 
Since  the  Information  Operations/  Sensors  topic  area  was  quite  broad,  it  was  expected  that 
the  chance  of  finding  a  suitable  target  program  from  this  domain  was  high.  However,  the 
primary  reason  for  the  lack  of  high  evaluation  score  was  the  current  lack  of  interest  in  the 
CART  technology  by  the  relevant  organizations  within  Electronic  Systems  Center  (ESC)  and 
Space  and  Missile  Center  (SMC),  and  the  lack  of  any  well-defined  constructive  modeling  and 
simulation  testbeds.  This  topic  area  will  also  be  revisited  during  the  Case  Study  2  evaluation. 

Clearly,  the  C4ISR  SoS  evaluated  very  highly  in  the  matrix.  It  was  discovered  that  the 
C4ISR  SoS  Virtual  Strike  Warfare  Environment  (VSWE),  which  is  the  modeling  and 
simulation  testbed  for  JSF,  provided  the  best  possible  solution  to  the  challenge.  It  offered  a 
seamless  transition  between  the  virtual  and  constructive  environments  —  consequently 
eliminating  the  problem  of  controlling  for  differences  in  simulation  results  unrelated  to  the 
HPM  vs.  the  human  operator  issue.  CART  also  seems  to  provide  a  clear  benefit  to  the  JSF 
program  in  terms  of  filling  a  stated  modeling  need,  as  the  JSF  program  has  already  emerged 
as  a  strong  CART  advocate.  In  addition,  the  JSF  modeling  and  simulation  program  has  a 
very  high  visibility  within  the  Air  Force  and  across  the  DoD.  Finally,  the  JSF  modeling  and 
simulation  program  is  currently  considered  a  leader  in  the  SBA  community,  and  the  CART 
program  should  benefit  greatly  from  this  affiliation.  No  other  topic  areas  could  match  these 
benefits. 

Recommendations 

The  recommendation  from  this  analysis  is  that  the  first  CART  case  study  topic  area  should  be 
the  JSF  C4ISR  SoS.  However,  all  other  topic  areas  considered  here  should  be  considered  for 
evaluation  again  during  the  search  for  a  topic  area  for  Case  Study  2.  The  next  section  of  this 
report  proposes  a  framework  for  integrating  a  CART  HPM  into  the  existing  C4ISR  SoS 
VSWE  modeling  and  simulation  testbed. 
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CART  INTEGRATION  FOR  CASE  STUDY  1 


Virtual  Strike  Warfare  Environment  (VSWE) 

The  current  modeling  and  simulation  testbed  in  place  for  JSF  is  the  VSWE.  Figure  A-l 
shows  the  basic  models  and  their  interrelationships.  The  different  modules  of  the  simulation 
environment  are  called  the  FRED,  the  SWEG,  and  the  S  WED  AT. 

FRED 

FRED  stands  for  the  Fighter  Requirements  Evaluation  Demonstrator  and  provides  the  virtual 
cockpit  management  software,  controls  the  avionics  and  aero  models,  and  manages  the 
controls,  displays,  and  out-the-window  scene  generation  software. 

SWEG 

SWEG  stands  for  Simulated  Warfare  Environment  Generator  and  is  the  mission-level 
modeling  environment.  It  is  an  event-stepped,  data-driven  modeling  environment  that 
provides  the  terrain,  targets,  threats,  and  all  C4ISR  data  and  events. 

SWEDAT 

SWEDAT  stands  for  SWEG  data  interface  and  is  an  area  in  shared  memory  that  allows  assets 
external  to  SWEG  to  dynamically  interact  with  SWEG.  The  SWEDAT  typically  resides  in  a 
SCRAMNET  shared  memory  interface. 

CART  HPM  Integration  into  the  VSWE 

A  proposed  architecture  for  integrating  a  CART  HPM  into  the  current  VSWE  environment  is 
shown  in  Figure  A-2.  The  current  VSWE  environment  is  shown  at  the  left,  the  CART  HPM 
is  shown  at  the  right,  and  a  proposed  High-Level  Architecture  (HLA)  Run-Time 
Infrastructure  (RTI)  is  shown  at  center,  linking  the  two  environments. 
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D  Represents  current  capability 
D  Represents  CART  team  effort 


Figure  A -2.  CART  HPM  Architecture  Integration  with  the  VSWE 
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APPENDIX  B 


TASK  NETWORK  DIAGRAMS  FOR  THE  CASE  STUDY  1  HUMAN 

PERFORMANCE  MODEL 
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will  cause  the  Determine  Jink  task  to  execute.  In  this  task,  a  new  desired  heading  will  be  calculated,  representing  a  pilot’s  desire  to 
jink  to  one  side  or  the  other  of  the  evasive  heading. 


effoit  to  evade  a  launched  missile.  It  consists  of  tasks  to  stow  the  TIR  if  deployed,  go  into  afterburner,  release  countermeasures,  and 
perform  a  manual  turn  to  the  evasive  heading.  Throughout  the  duration  of  the  evasive  turn,  heading,  attitude  and  altitude  are 
monitored. 


m 
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Description:  The  Navigate  Goal  Function  performs  both  acceptance  of  auto  re-plans  and  generation  of  manual  re-plans.  In  the  case  of 
manual  re-plans,  the  planner  mode  is  selected  and  then  the  plan  is  requested.  Types  of  manual  re-plans  include  planning  to  refly  the 
target  area,  planning  to  attack  an  identified  target  and  planning  to  abort. 
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function  includes  pilot  actions  to  update  the  target  location  in  the  mission  planner  once  the  target  update  is  received.  A  number  of 
dummy  tasks  are  used  to  check  for  the  update,  manage  workload,  and  to  clear  trigger  conditions. 


with  a  task  to  estimate  the  distance  to  the  target  area.  Next,  a  number  of  dummy  tasks  are  used  to  step  through  a  decision  process. 
These  include  ranking  the  objects  of  interest  based  on  their  range  to  the  reference  point  (i.e.,  the  best  estimate  of  the  target’s  location) 
and  whether  they  are  moving,  considering  alternate  lookpoints  for  both  the  SAR  and  TIR,  and  choosing  the  next  appropriate  object  off 
of  the  priority  list.  Next,  the  Choose  Sensor  and  Location  to  Image  operator  task  assigns  time  and  workload  to  the  decision  making 
task.  Dummy  tasks  are  also  included  to  end  the  Acquire  function  or  to  enable  imaging  the  target  as  appropriate. 


Dummy: 


Figure  B-22.  Task  Network  for  Image  Target  Function  Within  Acquire  Goal  Function 

Description:  Here,  the  tasks  required  for  sensor  employment  are  represented  They  include  stowing/deploying  the  TIR  as  necessary, 
selecting  the  radar,  TSD,  or  TIR  as  the  display  of  interest,  slewing  or  bumping  the  cursor  as  necessary  and  commanding  the  image. 


Johnson,  John  (1958).  Analysis  oflmage  Forming  Systems,  Image  Intensifier  Symposium,  Warfare  Vision  Branch,  Electrical  Engineering  Dept.  U.S.  Army  Engineer  Research  and  Development  Laboratories,  Ft  Belvoir  VA. 
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APPENDIX  C 


TASK  INFORMATION  REQUIREMENTS,  DECISION  DESCRIPTIONS,  AND 
RESULTING  COMMANDS  IDENTIFIED  IN  THE  MISSION  DECOMPOSITION 

PROCESS 
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Table  C-l.  Results  of  the  Mission  Decomposition 


<»oal 

Function  Name 

Task  Name 

Information  In 

Decision 

Command  Out  j 

Control  Aircraft  / 
Maintain  SA 

Monitor  Flight 
Instruments 

Check  Airspeed 

Current  Airspeed 

Control  Aircraft  / 
Maintain  SA 

Monitor  Flight 
Instruments 

Check  Altitude 

Current  Altitude 

Control  Aircraft  / 
Maintain  SA 

Monitor  Flight 
instruments 

Check  Heading 

Current  Heading 

t 

Control  Aircraft  / 
Maintain  SA 

Monitor  Flight 
Instruments 

Check  Attitude 

Current  Pitch  and 

Roll 

Control  Aircraft  / 
Maintain.  SA 

Monitor  TSD 

Check  Mission  Leg 

Current  Destination 
Point 

Control  Aircraft  / 
Maintain  SA 

Monitor  TSD 

Check  for  Threats 

Current  Number  of 
Launched  Missiles 

Control  Aircraft  / 
Maintain  SA 

Monitor  TSD 

Check  Range  and 
Bearing  to  Target 

Area 

Range  and  Bearing  to  ; 

Reference  Point 

* -,L' ,T*.,  \  l 

Control  Aircraft  / 
Maintain  SA 

Monitor  A/C 

Systems  Status 

Check  Fuel 

Lbs  Fuel  Remaining  J 

Control  Aircraft  / 
Maintain  SA 

Monitor  Audio 

Monitor  Audio 
Channel 

Launch  Tone  Present, 
Nav  Tone  Present, 
Comm  Message 
Present 

If  tones  are  present, 
trigger  appropriate 
;  goal  function  (Nav  or 
Evade).  If  Comm 
message  is  present, 
write  down  the 
message. 

Control  Aircraft  / 
Maintain  SA 

Monitor  Audio 

Receive/Copy 

Updated  Target 
Coordinates 

Updated  Target 
Lat/Lon 

Evade 

Monitor  Audio 

Monitor  Audio  ■  .  ,  ~  n 

,  i  Launch  Tone  Present 

Channel 

If  tones  are  present, 
trigger  appropriate 
goal  functions. 

Evade 

Execute  Evasion 
Strategy 

Initiate  Evade 
Maneuver 

Start  Evasive  Tum 

Evade 

Execute  Evasion 
Strategy 

Maintain  Evade 
Maneuver 

Current  heading, 
desired  evasive 
heading 

If  current  heading  = 
desired  evasive 
heading,  then  end  the 
lum.  Else,  continue 
to  turn. 

Continue  Evasive 

Turn 

Evade 

Execute  Evasion 
Strategy 

Release  Chaff/Flare 

Release  Chaff/Flare 

Evade 

Execute  Evasion 
Strategy  . . 

Retract  TIR 

Retract  TIR 

Evade 

Execute  Evasion 
Strategy 

Check 

Countermeasure 

Stores 

Chaff  Currently 
Available 

If  chaff/flare  is 
available,  release  it. 

Evade 

Execute  Evasion 
Strategy  . . ; 

Move  Throttle 

Move  Throttle  to 
Desired  Setting 

Evade 

Execute  Evasion 
Strategy 

Check  TIR 

TIR  Status 

If  TIR  is  deployed, 

|  then  retract  it. 

Evade 

Execute  Evasion 
Strategy 

Check  Heading 

Current  Heading 

:  If  current  heading  =  j 
;  desired  heading,  then 
;  end  the  turn.  Else, 
continue  to  turn.  ^ 

Evade 

Execute  Evasion 
Strategy 

Check  Attitude 

Current  Pitch  and 

Roll 

Evade 

Execute  Evasion 
Strategy 

Check  Altitude 

Current  Altitude 

Evade 

Execute  Evasion 
Strategy 

Abort  Evade 

Maneuver 

:  Roll  Out  at  Current 
?■  Heading 

125 


Table  C-l.  Results  of  the  Mission  Decomposition  (continued) 


Goal  Function  Name  Task  Name  Information  In  J 

Decision  Command  Out 

.  Determine  Maneuver  f  Current  Heading,  . 

Evade  i  5eleCt  Evasion  to  Intercept  Current  Bearing  to  | 

.•  Strategy  Minimization  Path  Threat  Emitter 

Calculate  heading  j 

that  will  steer  aircraft 

180  deg  away  from 
threat  emitter 
location. 

>■  \ 

Evaluate  Threat  :  Identify  Current  Current  Number  of  j 

Evade  Situation  Threat(s)  \  Launched  Missiles  j 

If  one  or  more  threats 
are  currently 
launched,  prioritize 
the  threats.  If  threat 
is  no  longer  active, 

re-engage  autopilot^  j- _ _ _ _ _ _ : 

Missile  Type,  Range  • 

Evade  Evaluate  Threat  prioritize  Threats  to  Emitter,  Current 

Situation  Aircraft  Altitude  J 

■  j 

Estimate  time  to 
intercept  for  all 
active  threats  and  j 

make  the  nearest 
threat  (in  time)  the 
highest  priority 

threat.  _ _ _ _ 

Evade  Re-Engage  Autopilot  i  Engage  Autopilot _ - - ! 

j  Engage  Autopilot 

:  .  .  .  ..  .  -Current  Autopilot 

Evade  Re-Engage  Autopilot  Check  Autopilot  $tatus 

If  autopilot  is  not 

engaged,  engage  it. . ,  _ _ _ _ _ _ _ _ _ _ 

Evade  Re-Engage  Autopilot  ?  Engage  Autothrottle  . . j 

Engage  Autothrottle 

Evade  Re-Engage  Autopilot  1  Move  Throttle  \ 

|:  Move  Throttle  to 
.1  [Desired  Setting 

'“"P  \  Plan  For  Reflying  Put  Cursor  on  Alt 

Navigate  Target  Area  ^  Point 

[.  Slew  Cursor  to 

Desired  Coordinates 

Plan  For  Refiying  Designate  as  Mustfly 

NaviSale  Target  Area  i  Point  . . ; 

^Designate  Current 
Cursor  Position 

?  Plan  For  Reflying  Put  Cursor  on 

Navigate  i  Target  Area  Release  Point  y 

Slew  Cursor  to 
l  Desired  Coordinates 

j  plan  For  Reflying  Select  PLAN  on  UFC 

Navigate  f  Target  Area  i  Planner  Page  . . 

|  j  Request  Re-plan 

k,  ■  ;  Plan  For  Reflying  pva|„9te  Re  n.an  Re-plan  is  Available, 

Navigate  i  Target  Area  Evaluate  Re-plan  Re.plan  Reason 

|  If  a  re-plan  is 
|  available  and  it  is  not 
|  returned  as  "error"  ! 

I  then  accept  it.  _ _  ■  .  . . 

:r  Plan  For  Refiying  A  R  , 

Navigate  Target  Area  Accept  Re-plan 

}  Accept  Re-plan 

T"'  -  rp Put  Cursor  Over 

Navigate  Plan  For  Attack  Target  Icon  _  : _ _ 

l  i  Slew  Cursor  to 

Desired  Coordinates 

_  A  .  Designate  as  Mustfly 

Navigate  Plan  For  Attack  pQ-nt 

P  '  Designate  Current 

■  Cursor  Position 

■  ,  „  ,  ,  Select  PLAN  on  UFC 

Navigate  Plan  For  Attack  planner  Pags _ _ 

Request  Re-plan 

■  ,  _  Re-plan  is  Available, 

Navigate  Plan  For  Attack  Evaluate  Re-plan  Re-plan  Reason 

)  If  a  re-plan  is 
;  available  and  it  is  not 
;  returned  as  "error" 

:  then  accept  it. _ _ _  :: . ^  r  _ _ _ _ = 

Respond  to  Auto  Re-  c  ,  D  ,  Re-plan  is  Available, 

Navigate  plaP  Evaluate  Re-plan  Re.pian  Reason 

Navigate  Respond  to  Auto  Re-  Accept  Re.plan 

Current  Display  of 

Navigate  Respond  to  Auto  Re-  Check  DQ1  Interest,  Desired 

P*an  Display  of  Interest 

Navigate  Respond  to  Auto  Re-  select  TSD  as  DOl 

Select  ABRT  on 

1  Navigate  Abort  ;  UFC  Planner  Page 

;  If  a  re-plan  is 
j  available  and  it  is  not 
j  returned  as  "error" 

then  accept  it.  . : 

Accept  Re-plan 

1  If  current  DOI  is  not 
the  desired  DOI, 

;  select  the  desired 

DOI.  .  . . . : . _ . „ . . . . . 

Set  TSD  as  Display 
■  of  Interest 

Request  Abort 
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Table  C-l.  Results  of  the  Mission  Decomposition  (continued) 


Coal 

Function  Name 

Task  Name 

Information  In  Decision 

Command  Out  "j 

Navigate 

r  Abort 

Evaluate  Re-plan 

Ufa  re-plan  is  3i 

l  Re-plan  is  Available,  ;  available  and  it  is  not  ] 

5  Re-plan  Reason  [  returned  as  "error" 

i  l  then  accept  it. 

Navigate 

_  ;  Abort 

I  Accept  Re-plan 

I  I 

;  Accept  Re-plan 

Navigate 

■  Select  Planner  Mode 
Sand  DPI  _ 

Select  Planner  Mode 
;  on  UFC 

5  \ 

1  Select  PLAN  on  UFC 

If  current  UFC  mode 

Navigate 

I  Select  Planner  Mode 
.  and  DOI 

:  Check  UFC  Mode 

Current  UFC  Mode  IS  n?t  de!j,red  ^FC 
mode,  select  the 

[ 

desired  UFC  mode. 

Navigate 

1=  Select  Planner  Mode 
\  and  DOI 

Check  DOI 

Current  Display  of  not 

t  Interest,  Desired  'he  desired  DOI 

Display  of  Interest  :^"he  desired 

f 

1 

Navigate 

Select  Planner  Mode 
and  DOI 

Select  TSD  as  DOI 

;  Select  TSD  as 

Display  of  Interest 

Acquire 

i  Evaluate  Image 

i  Detect  Objects 

j Emily type^siM. and 

moving  status  for  .  ,  ,  . 

r  i  Johnson  s  cntena  or 

i  objects  m  current  .  .  , 

,  J  r  ,  c  .  ;  GMTI  hit,  consider  it 

r  sensor  field  of  view 

f  detected 

; 

i  i  If  object  can  be 

■'  I  identified  based  on 

1  Johnson’s  criteria. 

Acquire 

\  Evaluate  Image 

{ Identify  Objects 

y  ~  ,  .  consider  it 

"identified".  If  object 
f  moving  status  for  ...  ._  . 

i  , .  .  is  identified  as  the 

?  objects  in  current 

•  *  r-  ,  j  r  -  t  target  of  interest, 

;  sensor  field  of  view  . & 

i  mgger  the 

li  ]!■ 

i 

appropriate  goal 
function  (Nav  or 
•  Attack). 

Acquire 

Update  Shootlist 

Check  DOI 

Current  Display  of  '[  c“rrent  P?^  not 

interest.  Desired  |  the  des.red  DOI 

p.  .  ,  n  .  *  select  the  desired 

%  Display  of  Interest  . 

j 

Acquire 

Update  Shootlist 

;  Check  NTS 

?  Current  Shootlist 
-Item  Selected 

j; 

Acquire 

Update  Shootlist 

Bump  NTS 

Step  Shootlist  Item 

Acquire 

■  Update  Shootlist 

Select  DOI 

1  Select  Desired  DOI 

Acquire 

1  Update  Shootlist 

Undesignate  to 
Remove  from 
Shootlist 

\  '1  ; 

Remove  Item  From 
Shootlist 

Acquire 

^  Update  Shootlist 

Place  Cursor  on 
|  Object  to  Add 

j  j 

j  Bump  Cursor  to 

Desired  Coordinates 

Acquire 

Update  Shootlist 

Designate  to  Add 
;  Detection  to  Shootlist 

■  Designate  Current 
Cursor  Position 

If  current  DOI  is  not  \ 

Acquire 

Designate  Target 

;  Check  DOI 

Current  Display  of  the  desired  DOI, 
Interest  i  select  the  desired 

i . . . . . . rpoi . 

Acquire 

Designate  Target 

Select  Radar  As  DOI  i 

vSelect  Radar  as 

Display  of  Interest 

Acquire 

r  Designate  Target 

;  Designate  as  Next  to 
Shoot 

Designate  Current 

Cursor  Position 

Acquire 

.  Designate  Target 

Slew  Cursor  to 

Target  Aimpoint 

Slew  Cursor  to 

Current  Target  of 

Interest  Position 

Acquire 

Designate  Target 

Select  TER  as  DO l 

Select  TIR  as  Display 
of  Interest 
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Table  C-l.  Results  of  the  Mission  Decomposition  (continued) 


Goal 

Function  Name 

Task  Name 

Information  In 

Decision  Command  Out  ! 

If  current  UFC  mode  it 

Acquire 

Update  Tgt  Location 

Check  UFC  Mode 

Current  UFC  Mode 

is  not  desired  UFC  | 
mode,  select  the 

desired  UFC  mode.  ,  .  J 

Acquire 

Update  Tgt  Location 

Select  Target  Mode 
on  UFC 

l  Select  Target  Mode 
_ j  on  UFC 

•Acquire 

Update  Tgt  Location 

Enter  Updated  Tgt 

La t/ Lon  in  UFC 

|;  Type  in  Updated 
l  Lat/Lon  in  UFC 

Acquire 

Update  Tgt  Location 

Accept  Updated  Tgt 
La t/ Lon  on  UFC 

Select  Accept  on 
r  UFC 

- - — . — 

—  - - - - 

■  If  reference  point  is 

Acquire 

1  Decide  Where/How 
i  to  Look 

Estimate  Range  and 
Bearing  to  Ref  Point 

Range  and  Bearing  to 
the  Reference  Point 

not  in  range  bearing 
to  TIR  or  SAR,  then  jf 
:  end  acquisition  j  . . 

| 

i 

If  in  TIR  range  and  C 
viable  objects  are  on  | 
the  shootlist,  select  * 

TIR,  next  Field  of 

Acquire 

( Decide  Where/How 
to  Look 

1  Choose  Sensor  and 
Location  to  Image 

Range  and  bearing  to 
the  reference  point 
for  all  shootlist  items. 
Moving  status  of  all 
shootlist  items. 

view,  and  highest  -i 

priority  object’s 
coordinates.  If  in 

1  TIR  range  and  no 
objects  on  shootlist, 
select  TER  and  next 
;  alternate  look  point. 

If  not  in  TIR  range, 
i  but  in  SAR 

mg/bearing  then 
:  select  radar  and  next  ;| 

:  radar  lookpoint. _ _  . ,  n . . . 

••  . — >«■•*'«*** 

If  next  image  is  TER 

Acquire 

Acquire 

Image  Target 

UmageTarget 

Check  TIR 

■ 

Current  TIR  Status 

and  TER  is  not 

deploved.  deploy  it.  . 

•  Deploy  TIR  _ _ _ ^ 

Acquire 

image  Target  _  . 

Retract  TIR  __ 

— - —  — - * 

Retract  TIR 

If  current  DOI  is  not 

Acquire 

Image  Target 

Check  DOI  for 
Designation 

Current  Display  of 
Interest 

j  the  desired  DOI, 

:  select  the  desired 

DOI. _ _ „„„ _ 

^  §eject  Appropriate 

DOI  (based  on  sensor 
;  i  chosen) _ _ 

Acquire 

Image  Target 

Select  DOI  for 
Designation 

Acquire 

Image  Target 

siew  Cursor  to  Alt 
Look  Location 

Bump  NTS 

j  Slew  Cursor  to 

Desired  Coordinates 

. - 

Step  Shootlist  Item  i 

Acquire 

Acquire 

lnidgc  i  <u  gwi 

Image  Target 

Designate  as  Next  to 
Shoot  _ _ _ 

Designate  Current 
l  Cursor  Position  j 

Acquire 

Image  Target 

Select 

FOV/Command 
:  Image _ 

:  Command  Image 

Acquire 

Acquire 

=  Image  Target 

Image  Target 

Check  UFC  Mode 

Select  Target  Mode 
on  UFC 

Current  UFC  Mode 

:  If  current  UFC  mode  3 
\  is  not  desired  UFC 
mode,  select  the 

i  desired  UFC  mode.  \ _ _ _ _ 

Select  Target  Mode 

j  . .  on  UFC  . . . ■ 

- . . * — ■ " 

—  - - - ..  - -  — 

. . . . " 

\  If  current  DOI  is  not 

Acquire 

Image  Target 

Check  DOI  for 
Imaging 

Current  Display  of 
Interest 

;  the  desired  DOI, 
select  the  desired 

doi.  . . . . . . . . J 

Acquire 

Image  Target 

*  Select  DOI  for 
Imaging 

Select  Appropriate 
DOI  (based  on  sensor 
chosen) 
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Table  C-l.  Results  of  the  Mission  Decomposition  (continued) 


Goal 

Function  Name 

Task  Name 

Information  In 

Decision 

Command  Out 

Acquire 

l  image  Target 

>  Slew  Cursor  to 
;  Object 

j  Slew  Cursor  to 

1  Desired  Location 

Acquire 

*  Check  TER 

Current  TIR  Status 

j  If  TER  is  deployed, 
retract  it. 

Acquire 

,4 _ _ 

__  [  Retract  TIR 

1 

?  Retract  TIR 

Attack 

1 

j  Monitor  Target 
i  Position  and  Range 

i  Check  LAR  on  TSD 

;  Range  and  Bearing  to 
f  Designated  Point 

\  If  target  is  within 
LAR,  initiate  weapon 
\  release  sequence. 

Attack 

Monitor  Target 
Position  and  Range 

[  Monitor  Tgt  Position 
j  in  TIR 

Range  and  Bearing  to 
Designated  Point 

!  If  target  is  within  t  1 

!  LAR,  initiate  weapon  \ 

release  sequence.  «  *  ! 

Attack 

i  Monitor  Target 
?  Position  and  Range 

Slew  TIR  to  Track 

Tgt 

Slew  Cursor  to 
Identified  Target  of 
Interest 

Attack 

( Monitor  Target 
j  Position  and  Range 

Check  DOI 

i  Current  Display  of 
f  Interest 

If  current  DOI  is  not 
the  desired  DOI, 
select  the  desired 

DOI. 

Attack 

Monitor  Target 
j  Position  and  Range 

Select  TIR  as  DOI 

Select  TIR  as  Display 
of  Interest 

Attack 

Monitor  Target 
[Position  and  Range 

Designate  as  Next  to 
Shoot 

Designate  Current 
Cursor  Position 

Attack 

i  Release  Weapon 

5  Command  Weapon 
j  Release 

Release  Weapon 

Attack 

\  Release  Weapon 

Slew  Cursor  to 

Target  Aimpoint 

j  j  Slew  Cursor  to 

\  Identified  Target  of 

. .  1  H  Interest 

Attack 

l  Release  Weapon 

|  Designate  as  Next  to  j 
■  Shoot 

j  Designate  Current 
il  Cursor  Position 

Attack 

Release  Weapon 

1  Check  DOI 

Current  Display  of  $ 
Interest 

If  current  DOI  is  not  = 
the  desired  DOI, 
select  the  desired 

DOI. 

Attack 

Release  Weapon 

j  Select  TIR  as  DOI  ? 

j 

;  Select  TIR  as  Display 

1  of  Interest 

Attack 

\  Release  Weapon 

Retract  TIR 

i 

Retract  TIR 

Attack 

;  Release  Weapon 

!  Monitor  Tgt  Position  :i 
in  TIR  l 

Range  and  Bearing  to  j 
Designated  Point 

If  target  is  within 

LAR,  release 
weapon. 

Attack 

Release  Weapon 

Slew  TER  to  Track 

Tgt  | 

!  Slew  Cursor  to 

Identified  Target  of 
:  Interest 

Attack 

Release  Weapon 

Re-Designate  to 

Track  . . . . . j 

1  Designate  Current 

Cursor  Position 

Attack 

’  Check  TIR 

Current  TIR  Status  >f™  is.retr5'ed’  i 

i  then  deploy  TIR. 
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APPENDIX  D 


TABLES  OF  PERFORMANCE  TIMES  AND  WORKLOAD  VALUES  ASSIGNED  TO 

MODEL  TASKS 
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Table  D-l.  Performance  Times  and  Workload  Values1  Assigned  to  Model  Tasks 


Goal 

Function 

Task 

Mean  Time 

V  A  C  P 

Acquire 

Decide  Where/How  to 

Choose  Sensor  and  Location  to 

dec  time/60; 

-  5.0  0.0  :  6.8  0.0 

Look 

Image 

Acquire 

Decide  Where/How  to 

Estimate  Range  and  Bearing  to 

0:00:01.00 

5.0  j  0.0  4.6  .  0.6 

______ 

Look 

Ref  Point 

III: 

Acquire 

Designate  Target 

Check  DOI 

0:00:00.50  ' 

■  4.0  S  6.0  1.0  0.0 

?  f  |  ; 

Acquire 

Designate  Target 

:  Designate  as  Next  to  Shoot 

i  0:00:00.40 

O.o’  0.0  1.6  2.2 

Acquire 

Designate  Target 

Select  Radar  As  DOI 

]  0:00:00.90 

!  1.0  0.0  f.0 *  2.2 

Acquire 

Designate  Target 

Select  TIR  as  DOI 

0:00:00.90 

1.0  0.0  1.0  f  2/2 

Acquire 

Designate  Target 

Slew  Cursor  to  Target  Aimpoint 

:  0:00:02.00 

5.4  0.0  1.0  5.8  ' 

Acquire 

Evaluate  Image 

Detect  Objects 

0:00:02.00 

:  7.0  j  0.0  f  3.7  0.0 

Acquire 

Evaluate  Image 

Identify  Objects 

0:00:02.00 

l  7.0  ?  0.0  \  6.8  0.0 

Acquire 

Image  Target 

Bump  NTS 

0.6*(pl_number/2)/60; 

4.0  0.0  j  1.2  j  2.2 

Acquire 

Image  Target 

Check  DOI  for  Designation 

,  0:00:00  50 

40  :  oFT  L0  0.0 

Acquire 

Image  Target 

Check  DOI  for  Imaging 

:  0:00:00.50 

4.0  0.0  1.0  0.0 

Acquire 

Image  Target 

Check  TIR 

|  0:00:00.50 

T  4.o  Tool  TFT  5F 

Acquire 

Image  Target 

Check  UFC  Mode 

j  0:00:00.70 

\  4.6J6F)  TF1  oF 

Acquire 

Image  Target 

Deploy  TIR 

/  0:00:00.90 

,  4.0  ao'T  76  2.2 

Acquire 

Image  Target 

Designate  as  Next  to  Shoot 

;  0:00:00.40 

0.0  .  0.0  i  1.0  2.2 

s  : 

Acquire 

Image  Target 

Retract  TIR 

!  0:00:00.90 

4.0  :  0.0  ’1.0  2.2 

Acquire 

Image  Target 

Select  DOI  for  Designation 

0:00:00.90 

1.0  7571/)  1  'll 

Acquire 

Image  Target 

Select  DOI  for  imaging 

0:00:66.90 

t if’  ojvirir 

Acquire 

image  Target 

Select  FOV/Command  Image 

;  task_time/60; 

iF  aoT  Tol  53T 

Acquire 

Image  Target 

Select  Target  Mode  on  UFC 

;  0:00:01.08 

i.o  ox) ixTT 2 3T 

Acquire 

Image  Target 

Slew  Cursor  to  Alt  Look  Location 

.. 

Acquire 

Image  Target 

Slew  Cursor  to  Object 

1  0:00:02.00 

5.4  6.0  T.  0  5^8 

Acquire 

Update  Shootlist 

Bump  NTS 

0:00:00.60 

4i0"  r^oTTFf  2F 

Acquire 

Update  Shootlist 

Check  DOI  . 

i  0:00:00.50 . 

4.0  0.0 :  i.o  0.6 

Acquire 

Update  Shootlist 

Check  NTS 

0:00:00  50 

3 

4.0  'b.6  *  1.2  6F 

Acquire 

Update  Shootlist 

Designate  to  Add  Detection  to 

7  0:00:00.40 

Shootlist 

l 

ill: 

Acquire 

Update  Shootlist 

Place  Cursor  on  Object  to  Add 

0:00:02.00 

;  5.0  6.6 . TToTSF 

Acquire 

Update  Shootlist 

Select  DOI 

0:00:00.90 

f  1.0  65  i.o  f  I2 

Acquire 

Update  Shootlist 

Undesignate  to  Remove  from 

;  0:00:00  40 

•  3  } 

6.0  ’  0.0  1.0  .  2.2 

Shootlist 

f  t 

Acquire 

Update  Tgt  Location 

Accept  Updated  Tgt  Lat/Lon  on 

0:00:00.40 

I.O  6.0  1.0  !  2.2 

UFC 

•  t  1 

Acquire 

Update  Tgt  Location 

Check  UFC  Mode 

0:00:00.70 

4.6  oF  To  '  6.0 

Acquire 

Update  Tgt  Location 

Enter  Updated  Tgt  Lat/Lon  in 

'*■  0:00:15.00 

5.9  OX)  * v  5.3  7?0 

UFC 

Acquire 

Update  Tgt  Location 

Select  Target  Mode  on  UFC 

0:00:01.08 

:  1.0  651  1.0  2.2 

Acquire 

1.  i  /  a  /^r» _ . 

Check  TIR 

0:00:00.50 

•  4.0  6.0  L0  0.0 

1  VACP  values  in  table  represent  visual,  auditory,  cognitive  and  psychomotor  dimensions,  respectively. 
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Table  D-l.  Performance  Times  and  Workload  Values  Assigned  to  Model  Tasks 

(continued) 


Goal 

Function 

Task 

Mean  Time 

V 

4.0  ; 

A  (  C  j,P  | 

Acquire  j  Check  T1R  (MXWWJO  _  _  j 

i.v 

Acquire 

Deploy  TIR 

0:00:00.90 

4.0 

0.0 

1.0 

2.2 

Acquire 

Retract  TIR 

0:00:00.90 

0.0 

0.0 

1.0 

2.2 

Attack 

Monitor  Target 

Position  and  Range 

Check  DO I 

0:00:00.50 

4.0 

0.0 

1.0 

0.0 

Attack 

Monitor  Target 

Position  and  Range 

Check  LAR  on  TSD 

0:00:00.70 

5.0 

0.0 

4.6 

0.0 

Attack 

Monitor  Target 

Position  and  Range 

Designate  as  Next  to  Shoot 

0:00:00.40 

0.0 

0.0 

1.0 

2.2 

Attack 

Monitor  Target 

Position  and  Range 

Monitor  Tgt  Position  in  TER 

0:00:00.70 

4.0 

0.0 

1.0 

0.0 

Attack 

Monitor  Target 

Position  and  Range 

Select  TIR  as  DOI 

0:00:00.90 

1.0 

0.0 

1.0 

2.2 

Attack 

Monitor  Target 

Position  and  Range 

Slew  TIR  to  Track  Tgt 

0:00:02.00 

5.4 

0.0 

1.0 

5.8 

Attack 

Release  Weapon 

Check  DOI 

0:00:00.50 

4.0 

0.0 

1.0 

0.0 

Attack 

Release  Weapon 

Command  Weapon  Release 

0:00:00.40 

0.0 

0.0 

1.0 

2.2 

Attack 

Release  Weapon 

Designate  as  Next  to  Shoot 

0:00:00.40 

0.0 

0.0 

1.0 

2.2 

Attack 

Release  Weapon 

Monitor  Tgt  Position  in  TIR 

0:00:00.70 

4.0 

0.0 

1.0 

0.0 

Attack 

Release  Weapon 

Re-Designate  to  Track 

0:00:00.40 

0.0 

0.0 

1.0 

2.2 

Attack 

Release  Weapon 

Retract  TIR 

0:00:00.90 

0.0 

0.0 

1.0 

2.2 

Attack 

Release  Weapon 

Select  TIR  as  DOI 

0:00:00.90 

1.0 

0.0 

1.0 

2.2 

Attack 

Release  Weapon 

Slew  Cursor  to  Target  Aimpoint 

0:00:02.00 

5.4 

0.0 

1.0 

5.8 

Attack 

Release  Weapon 

Slew  TIR  to  Track  Tgt 

0:00:02.00 

5.4 

0.0 

1.0 

5.8 

Evade 

Evaluate  Threat 

Situation 

Identify  Current  Threat(s) 

0:00:00:70 

5.9  j 

0.0 

5.3 

0.0 

Evade 

Evaluate  Threat 

Situation 

Prioritize  Threats 

(0.3 1/60)  *(tht_count  * 

(tht  count- 1)/2); 

5.0 

0.0 

7.0 

0.0 

Evade 

Execute  Evasion 

Strategy 

Check  Altitude 

0:00:00.50 

5.9 

0.0 

3,7 

0.0 

Evade 

Execute  Evasion 

Strategy 

Check  Attitude 

0:00:00.50 

5.0 

0.0 

1.0 

0.0 

Evade 

Execute  Evasion 

Strategy 

Check  Countermeasure  Stores 

0:00:00.70 

5.9 

0.0 

3.7 

0.0 

Evade 

Execute  Evasion 

Strategy 

Check  Heading 

0:00:00.50 

5.0 

0.0 

4.6 

0.0 

Evade 

Execute  Evasion 

Strategy 

Check  TIR 

0:00:00.50 

4.0 

0.0 

1.0 

0.0 

Evade 

Execute  Evasion 

Strategy 

Initiate  Evade  Maneuver 

0:00:00.50 

5.4 

0.0 

4.6 

2.6 

Evade 

Execute  Evasion 

Strategy 

Maintain  Evade  Maneuver 

0:00:00.50 

0.0 

0.0 

0.0 

2.6 

Evade 

Execute  Evasion 
Strategy 

Move  Throttle 

0:00:00.50 

0.0 

0.0 

1.0 

5.8 

Evade 

Execute  Evasion 
Strategy 

Release  Chaff/Bare 

0:00:00.40 

0.0 

0.0 

1.0 

2.2 

Evade 

Execute  Evasion 
Strategy 

Retract  TIR 

0:00:00.90 

0.0 

0.0 

1.0 

2.2 

Evade 

Monitor  Audio 

Monitor  Audio  Channel 

0:00:00.50 

0.0 

1.0 

1.0 

0.0 

Evade 

Re-Engage  Autopilot 

Check  Autopilot 

0:00:00.70 

4.0 

0.0 

1.0 

0.0 

Evade 

Re-Engage  Autopilot 

Engage  Autopilot 

0:00:01.40 

0.0 

0.0 

1.0 

2.2 

Evade 

Re-Engage  Autopilot 

Engage  Autothrottle 

0:00:01.40 

0.0 

0.0 

1.0 

2.2 
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Table  D-l.  Performance  Times  and  Workload  Values  Assigned  to  Model  Tasks 

(continued) 


Goal 

Evade 

Function 

Re-Engage  Autopilot 

Task 

Move  Throttle 

Mean  Time 

0:00:01.00 

V 

5.0 

A 

0.0 

c 

1.0 

p 

5  8 

Evade 

Select  Evasion 

Strategy 

Determine  Jink 

0:00:00.07 

5.0 

0.0 

1.2 

Evade 

Select  Evasion 

Strategy 

Determine  Maneuver  to  Intercept 
Minimization  Path 

0:00:00.07 

5.0 

0.0 

6.8 

0.0 

Mission 

Level 

Monitor  A/C  Systems 
Status 

Check  Fuel 

0:00:00.70 

5.9 

0.0 

3.7 

0.0 

Mission 

Level 

Monitor  Audio 

Monitor  Audio  Channel 

0:00:00.50 

0.0 

1.0 

Mission 

Level 

Monitor  Audio 

Receive/Copy  Updated  Target 
Coordinates 

0:00:23.00 

5.9 

H 

■ 

Mission 

Level 

Monitor  Flight 
Instruments 

Check  Airspeed 

0:00:00.70 

0.0 

Mission 

Level 

Monitor  Flight 
Instruments 

Check  Altitude 

0:00:00.50 

■ 

0.0 

3.7 

Mission 

Level 

Monitor  Flight 
Instruments 

Check  Attitude 

0:00:00.50 

5.0 

0.0 

Mission 

Level 

Monitor  Flight 
Instruments 

Check  Heading 

0:00:00.50 

0.0 

0.0 

Mission 

Level 

Monitor  TSD 

Check  for  Threats 

0:00:00.50 

4.0 

0.0 

Mission 

Level 

Monitor  TSD 

Check  Mission  Leg 

0:00:00.70 

4.0 

0.0 

1.0 

0.0 

Mission 

Level 

Monitor  TSD 

Check  Range  and  Bearing  to 

Target  Area 

0:00:01.00 

5.0 

HI 

0.0 

Abort 

Accept  Re-plan 

0:00:01.80 

0.0 

0.0 

1.0 

2.2 

Abort 

Evaluate  Re-plan 

0:00:01.27 

7.0 

■gl 

0.0 

Navigate 

Abort 

Select  ABRT  on  UFC  Planner 

Page 

0:00:01.08 

1.0 

2.2 

Navigate 

Plan  For  Attack 

Accept  Re-plan 

0:00:01.80 

1.0 

0.0 

1.0 

'IggH 

Navigate 

Plan  For  Attack 

Designate  as  Mustfly  Point 

0:00:00.40 

0.0 

ill 

Navigate 

Plan  For  Attack 

Evaluate  Re-plan 

0:00:01.77 

7.0 

0.0 

0.0 

Navigate 

Plan  For  Attack 

Put  Cursor  Over  Target  Icon 

0:00:02.00 

5.0 

5.8 

Navigate 

Plan  For  Attack 

Select  PLAN  on  UFC  Planner 

Page 

0:00:01.08 

1.0 

0.0 

Navigate 

Plan  For  Reflying 

Target  Area 

Accept  Re-plan 

0:00:01.80 

1.0 

0.0 

1.0 

2.2 

Navigate 

Plan  For  Reflying 

Target  Area 

Designate  as  Mustfly  Point . 

0:00:00.40 

0.0 

Navigate 

Plan  For  Reflying 

Target  Area 

Evaluate  Re-plan 

0:00:01.77 

a 

0.0 

Navigate 

Plan  For  Refiying 

Target  Area 

Put  Cursor  on  Alt  Point 

0:00:02.00 

5.0 

0.0 

1.0 

5.8 

Navigate 

Plan  For  Reflying 

Target  Area 

Put  Cursor  on  Release  Point 

0:00:02.00 

5.0 

0.0 

1.0 

5.8 

Navigate 

Plan  For  Reflying 

Target  Area 

Select  PLAN  on  UFC  Planner 

Page 

0:00:01.08 

1.0 

0.0 

1.0 

2.2 

Navigate 

Respond  to  Auto  Re- 
!  plan 

Accept  Re-plan 

0:00:00.40 

1.0 

0.0 

1.0 

2.2 

Navigate 

Respond  to  Auto  Re¬ 
plan 

Check  DOl 

0:00:00.50 

4.0 

0.0 

1.0 

0.0 

Navigate 

Respond  to  Auto  Re¬ 
plan 

Evaluate  Re-plan 

0:00:01.77 

7.0 

0.0 

6.8 

0.0 

Navigate 

Respond  to  Auto  Re¬ 
plan 

Select  TSD  as  DOI 

0:00:00.90 

1.0 

0.0 

1.0 

2.2 

Navigate 

Select  Planner  Mode 
and  DO I 

Check  DOI 

0:00:00.50 

4.0 

0.0 

1.0 

0.0 
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Table  D-l.  Performance  Times  and  Workload  Values  Assigned  to  Model  Tasks 

(continued) 


Goal 

Navigate 

Function 

Select  Planner  Mode 
and  DOI 

Task 

Check  UFC  Mode 

Mean  Time 

0:00:00.70 

v 

4.0 

A 

0.0 

c 

1.0 

0.0 

Navigate 

Select  Planner  Mode 
and  DOI 

Select  Planner  Mode  on  UFC 

0:00:01.08 

1.0 

0.0 

1.0 

2.2 

Navigate 

Select  Planner  Mode 
and  DOI 

Select  TSD  as  DOI 

0:00:00.90 

1.0 

0.0 

1.0 

2.2 
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VARIABLES  USED  IN  CASE  STUDY  1  HUMAN  PERFORMANCE  MODEL 
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Table  E-l.  Variables  Used  in  Case  Study  1  Human  Performance  Model 


Variable  Name  Variable  Type  External 


abort_done 

absjurn 

add_sl_count 

AFTERBURNER 

alt_ptjndex 

alt_pt_inuse 

alt_pt_used 

alt_pts_done 

atk_rpln_don 

attack_done 

audio_done 

auto_pilot 

AUTOTHROTTLE 

bad_tht 

check_upd 

cmd_snsr_cl 

cns_moving 

count 

CRUISE 

ct 

cur_throttle 

curr_emitter 

current_doi 

dec_time 

deLsLcount 

detect_il 


end_acq 

end_audio 

end_evd_wkld 

end_turn 

evai_gnnt_hit 

eval_moving 

eval_mg 

evaLsize 

evaLtype 

found 

found„ptr 

found_status 

FULL.MIL 

give.up 

halt.done 

id_to_image 

ident_as_tgt 

idx 

in_sar_rng 

in_snsr_rng 
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Table  E-l.  Variables  Used  in  Case  Study  1  Human  Performance  Model  (continued) 


Variable  Name  I  Variable  Type  External 


in_tir_rng 

index 

index_to_alt 

index_to_pl 

index_wrap 

input_brg 

inputjljdx 

input_pijdx 

input_rng 

input_sensor 

int_min_hdg 

K_A_PILOT_D 

K_A_PILOT_E 

K_A_THROTTLE 

K_ALT_LK_LAT 

K_ALT_LK_LON 

K_ALT_PT_LAT 

K_ALT_PT_LON 

K_BINGO_FUEL 

K_BLD_AIRFLD 

k_bld_bridge 

K_BLD_COM 

K_BLD_MFG 

K_BLD_PWR 

K_BLD_SAM 

K_CALC_ALTPT 

K_CURSOR_AC 

K_DELAY_END 

K_DEPLOY_TIR 

K_DESIG_PT 

KJDETECTED 

K_DFLT_SNSR 

K_DOLC  ENTER 

K„DOI_CTRMPD 

K.DOLLEFT 

K.DOLLFTMPD 

K„DOI_RIGHT 

K_DOI_RTMPD 

K_EMT_TYPE_1 

K_EMT_TYPE_2 

K_EVADE„MNVR 

K_F_ACP_RPLN 

K_GAMEOVER 

K_GENJMAGE 

K_GND_BRDM 


Integer 

Integer 

Integer 

Integer 

Integer 

Real 

Integer 

Integer 

Real 

Integer 

Real 

Integer 

Integer 

Integer 

Array  of  Reals 

Array  of  Reals 

Array  of  Reals 

Array  of  Reals 

Real 

Integer 

Integer 

Integer 

Integer 

Integer 

Integer 

Integer 

Integer 

Integer 

Integer 

Integer 

Integer 

Integer 

Integer 

Integer 

Integer 

Integer 

integer 

Integer 

Integer 

Integer 

Integer 

Integer 

Integer 

Integer 

Integer 
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Table  E-l.  Variables  Used  in  Case  Study  1  Human  Performance  Model  (continued) 


Table  E-l.  Variables  Used  in  Case  Study  1  Human  Performance  Model  (continued) 


Array  of  Integers 
Integer 

Array  of  Integers 

Real 

Real 

Real 

Integer 

Array  of  Integers 
Integer 

Array  of  Integers 
Array  of  Reals 
Array  of  Reals 


Variable  Name  Variable  Type 


K_R  P_TY  P_R  EQ 
K_RP_TYP_THT 
K_S_LPJDX 
K_S_LP_MAX 
K_S_LP_SNSR 
K_SAR_GIM_HI 
K_SAR_GIM_LO 
K_SAR_RNG 
K_STOW_TIR 
K_T_LPJDX 
K_T_LP_MAX 
K_T_LP_SNSR 
K_TGTBEG_LAT 
K_TGTBEG_LON 
K_THROTTLE 
K_TI  R_2X_N  AR 
K_TI  R_NARROW 
K_TIR_RNG 
K_TIR_WIDE 
K_U_ACP_RPLN 
K_UFC_LL_ACP 
K_UFC_NAV 
K_UFC_PLAN 
K_UFC_PLNMOD 
K_UFC_TGT 
K_UFC_TGT_LL 
;K_UFC_TGTMOD 
K_UFC_THT 
K_UNVIABLE 
K_UPD_REFLOC 
K_UPD_SENSOR 
K_WSAR_HIGH 
K_W  S  AR_LOW 
K_WSAR_MED 
L_LK_PT_LAT 
l_num_iar 
last_apd_cnt 
lasLrtiJrg 
last_sensor 
last_snsr_cl 
lastjirjd 
lastpLcount 
latjojmage  Real 

!on__to_jmage  Real 

must_evade  Integer 


External 


FALSE 

FALSE 

FALSE 

FALSE 

FALSE 

FALSE 

FALSE 

FALSE 

FALSE 

FALSE 

FALSE 

FALSE 

FALSE 

FALSE 

FALSE 

FALSE 

FALSE 


Table  E-l.  Variables  Used  in  Case  Study  1  Human  Performance  Model  (continued) 


Variable  Name 

Variable  Type 

External 

nav_abrt_pdg 
navjnd  pdg 

Integer 

Integer 

■KgjUSH 

ISIS’' 

nav_nfnd_pdg 

Integer 

FALSE 

need_navplan 

Integer 

FALSE 

need_planchk 

Integer 

FALSE 

need_reflook 

Integer 

FALSE 

next_doi 

Integer 

FALSE 

nextjsarjp 

Integer 

FALSE 

nextjtirjp 

Integer 

FALSE 

nextjink 

Integer 

FALSE 

next_usar_lp 

Integer 

FALSE 

next_utirjp 

Integer 

FALSE 

nxtsnsr 

Integer 

FALSE 

obj_ang!e 

Real 

FALSE 

obj_detected 

Integer 

FALSE 

objjdent 

Integer 

FALSE 

objJs_toi 

Integer 

FALSE 

OFF 

Integer 

FALSE 

ON 

Integer 

FALSE 

orig_wrp_act 

Integer 

FALSE  ■ 

orig_wrp_id 

Integer 

FALSE 

p_ac_lat 

Real 

FALSE 

p_ac_lon 

Real 

FALSE 

p_airspeed 

Real 

FALSE 

p_altitude 

Real 

FALSE 

P_brg_to_dp 

Real 

FALSE 

P_brg_to_ref 

Real 

FALSE 

p_chf_avail 

Integer 

FALSE 

p_curr_doi 

Integer 

FALSE 

p_dest_lat 

Real 

FALSE 

p_dest_lon 

Real 

FALSE 

p_flr_avail 

Integer 

FALSE  • 

pJueLqty 

Real 

FALSE 

p_heading 

Integer 

FALSE 

pjaunchjon 

Integer 

FALSE 

p.pitch 

Real 

FALSE 

p_power 

Real 

FALSE 

P_rng_to_dp 

Real 

FALSE 

P_mg_to_ref 

Real 

FALSE 

p_roll 

Real 

FALSE 

p_rp*n— avail 

Integer 

FALSE 

p„rpln_reasn 

Integer 

FALSE 

P_rpln_ton 

Integer 

FALSE 

P  Jgt_update 

Integer 

FALSE 
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Table  E-l.  Variables  Used  in  Case  Study  1  Human  Performance  Model  (continued) 


Variable  Name 

Variable  Type 

External 

p_tht_count 

Integer 

FALSE 

p_tht_emtbrg 

Array  of  Reals 

FALSE 

p_tht_emtrng 

Array  of  Reals 

FALSE 

p_tht_emttyp 

Array  of  Integers 

FALSE 

p_tht_msltti 

Array  of  Reals 

FALSE 

p_ufc_mode 

Integer 

FALSE 

pass 

Integer 

FALSE 

pass^clock 

Real 

FALSE 

pUn_mg 

Integer 

FALSE 

plist_count 

Integer 

FALSE 

pre_tir_pndg 

Integer 

FALSE 

pre_tir_upd 

Integer 

FALSE 

prev^emitter 

Integer 

FALSE 

prevjmgjd 

Integer 

FALSE 

prLnumber 

Integer 

FALSE 

ptr 

Integer 

FALSE 

px_displayed 

Real 

FALSE 

px_to_detect 

Real 

FALSE 

pxjojdent 

Real 

FALSE 

recording 

Integer 

FALSE 

ref_adjust 

Integer 

FALSE 

refjn_rng 

Integer 

FALSE 

replanjimer 

Real 

FALSE 

rp_next_step 

Integer 

FALSE 

SAR 

Integer 

FALSE 

sensor 

Integer 

FALSE 

shortest_tti 

Real 

FALSE 

shotdown 

Integer 

FALSE 

slew_needed 

Integer 

FALSE 

startjimer 

Real 

FALSE 

tan_angle 

Real 

FALSE 

taskjime 

Real 

FALSE 

tempjidg 

Real 

FALSE 

tgt_recorded 

Integer 

FALSE 

tgt_upd_done 

Integer 

FALSE 

thtjndex 

Integer 

FALSE 

TIR 

Integer 

FALSE 

tir_deployed 

Integer 

FALSE 

tirjov 

Integer 

FALSE 

toijound 

Integer 

FALSE 

toijndex 

Integer 

FALSE 

top_threat 

Integer 

FALSE 

trg_acq_rb 

Integer 

FALSE 

trg_acq_upd 

Integer 

FALSE 
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Table  E-l.  Variables  Used  in  Case  Study  1  Human  Performance  Model  (continued) 


Variable  Name  Variable  Type  External 


trg_attack 

trg_evade 

trg_nav_abrt 

trg_nav_fnd 

trg_nav_nfnd 

trg_nav_pln 

trg_nav_rte 

turn 

ufc_mode 

upd_refjat 

upd_ref_lon 

upd_wrpjat 

upd_wrp_lon 

WKLD^ACQ 

WKLD.ATK 

WKLD_FTP 

WKLD_NAV 

wpn_clock 

wpt_tmpclock 

zoom_onty 

FALSE 

TRUE 

ac_airspeed 

ac_altitude 

ac_fueLqty 

ac_heading 

ac_lat 

acjon 

ac_pitch 

ac_power 

ac_rol! 

ac_shotdown 

apd_brg 

apd_cnt 

apd_rng 

brg_to_dp 

brg_to_ref 

chf_avail 

desig_e!ev 

desig„tat 

desig_lon 

dest_pt_id 

dest_pt_lat 

desLptJon 

flr_avail 


Integer 

Integer 

Integer 

Integer 

Integer 

Integer 

Integer 

Real 

Integer 

Real 


Real 

Integer 

Integer 

Integer 

Integer 

Real 

Real 

Integer 

Integer 

Integer 

Rea! 


Integer 

Real 

Integer 

Real 

Integer 

Real 

Real 

Real 

Integer 

Real 

Real 

Real 

Integer 

Real 

Real 

Integer 
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iLtype 

launchjon 

nnp„id 

mp_idx_2_tbm 


Integer 

Real 

Real 

Array  of  Reals 


Table  E-l.  Variables  Used  in  Case  Study  1  Human  Performance  Model  (continued) 


Variable  Name  Variable  Type  External 


hpm_altjat 

hpm_altJon 

hpm_cmdjd 

hpm_curs_lat 

hpm_cursJon 

hpm„des_hdg 

hpm„des_thtl 

hpm_des_tum 

hpm_refjat 

hpm_refJon 

hpm_rtLtrig 

hpm_updjat 

hpm_upd_lon 

ar_elev 

ar_num_det 

ar_numjd 

ar_num_ob] 

ar__obj_jd 

ar_obj_lat 

ar^objjon 

ar_obj_mvng 

ar_obj_nxts 

ar_obj_stat 

ar_rtljrig 

ar_sensor 

Lelev 

Lgmti_det 

IJmageJd 

IJat 

IJon 

Lmoverjdx 

Lmoving 

Lnumber 

Lobjjd 

l_prev_det 

Lprevjd 

Lrange 

Lrng_ref 


Array  of  Integers 
Array  of  Reals 
Array  of  Reals 
Array  of  Integers 
Array  of  Integers 
Array  of  Integers 


Array  of  Reals 
Array  of  Integers 
Integer 

Array  of  Reals 
Array  of  Reals 
Array  of  Integers 
Array  of  Integers 
Integer 

Array  of  Integers 
Array  of  Integers 
Array  of  Integers 
Array  of  Reals 
Array  of  Reals 
Integer 

Array  of  Reals 
Array  of  Integers 
Integer 

Array  of  Integers 
Integer 
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Table  E-l.  Variables  Used  in  Case  Study  1  Human  Performance  Model  (continued) 
Variable  Name  Variable  Type  External 


mp_num 

obj_mg 

pl_brg_ac 

pLcount 

pl_Id 

pljat 

pljon 

pLnum_add 

pl_num_del 

pl_number 

pLnxtsensor 

pl_mg_ac 

pl_mg„ref 

pl_type 

pl_viable 

mg_to_dp 

mg„to_ref 

route 

rpln_avail 

rpln_count 

rpln_reason 

rpln_ton 

sen$or_2_use 

tgt_update 

tht_count 

tht_emtbrg 

tht_emtid 

tht_emtmg 

tht_emttyp 

tht_mslid 

tht_m$ltti 

tir_dep_stat 

tir_trk_stat 

updated_lat 

updatedjon 


Array  of  Reals 
Array  of  Reals 
Integer 

Array  of  Reals 
Array  of  Reals 
Integer 

Array  of  Integers 
Array  of  Reals 
Array  of  Reals 


Array  of  Reals 
Array  of  Integers 
Array  of  Reals 
Array  of  Integers 
Array  of  Integers 
Array  of  Reals 
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APPENDIX  F 


MODEL  CODE  RELEASE  CONDITIONS,  BEGINNING  EFFECTS,  ENDING 
EFFECTS,  AND  DECISION  NODES 
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Table  F-l.  Model  Code  Release  Conditions,  Beginning  Effects,  Ending  Effects,  and  Decision  Nodes 
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Table  F-l.  Model  Code  Release  Conditions,  Beginning  Effects,  Ending  Effects,  and  Decision  Nodes  (continued) 
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Table  F-l.  Model  Code  Release  Conditions,  Beginning  Effects,  Ending  Effects,  and  Decision  Nodes  (continued) 
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Table  F-l.  Model  Code  Release  Conditions,  Beginning  Effects,  Ending  Effects,  and  Decision  Nodes  (continued) 
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Table  F-l.  Model  Code  Release  Conditions,  Beginning  Effects,  Ending  Effects,  and  Decision  Nodes  (continued) 
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Table  F-l.  Model  Code  Release  Conditions,  Beginning  Effects,  Ending  Effects,  and  Decision  Nodes  (continued) 
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Table  F-l.  Model  Code  Release  Conditions,  Beginning  Effects,  Ending  Effects,  and  Decision  Nodes  (continued) 


Table  F-l.  Model  Code  Release  Conditions,  Beginning  Effects,  Ending  Effects,  and  Decision  Nodes  (continued) 


Table  F-l.  Model  Code  Release  Conditions,  Beginning  Effects,  Ending  Effects,  and  Decision  Nodes  (continued) 
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Table  F-l.  Model  Code  Release  Conditions,  Beginning  Effects,  Ending  Effects,  and  Decision  Nodes  (continued) 


Table  F-l.  Model  Code  Release  Conditions,  Beginning  Effects,  Ending  Effects,  and  Decision  Nodes  (continued) 


04 

OO 


Table  F-l.  Model  Code  Release  Conditions,  Beginning  Effects,  Ending  Effects,  and  Decision  Nodes  (continued) 


p_curr_doi<>K_D  Select  TIR  as 
OLRIGHT  DOI 


Table  F-l.  Model  Code  Release  Conditions,  Beginning  Effects,  Ending  Effects,  and  Decision  Nodes  (continued) 


«  S 
^  e  £  © 

H  £  £  55 


■  *3s 

e 

o 


1 


& 

H1 


I 

W 


Xt 

e 

W 


I 

& 

t 

■g 

■  C  ■ 

I5 


•s 


s  5  « 


3  if 
i  os 


i  o 

!  £3  _  - 
&  u  .£  ' 
:  U  S?  O 

1^1 

w  2  <  , 


os 

P 


-  2 
co  H 


S  5 


>>H 

5 

S  i= 

8  k 

*  a 
7  tf 


£  w  ;  £ 


°  g. 

o  > 


•-  9,  5 


^  St  «.  = 
5  y  ; 
J5  o.i 
.£  o  > 


J  +  O  ll, 

gf3i 

id1  'c  ¥  t1 

|  c'i  v 


o 

H 


x  @ 


c  E-  -a,  ; 


S  Q  g  X  J  I 
o.  a.  c  5  h  : 

E's  P.|  g; 

.©"OS  II  -J  O  : 


r-  -r 

7  J  _*  <  2 

II  *  7f  U.  OS 

“O  .ll.  1  II  f- 

H2  00  T3  II. 
o  I  *u  «  O 

f 

4  5  s'  l=,  s'-.is' 

~  £  j5*£  2*.=  2 


V 

—  x 


~  .  y 
n  07  T3  != 


0  0-0 
c«  tr  is 


«  _  .2  x 


CJ 

e'<* 

J© 
2  'I 

^  A 


> 

O 

3 

ii. 

■6 


»  tr 

I  t 

+  a> 


:  — ■  £  o.  y  . 


.  ^  £  i 

i -3*8  5  I 


*3 


■g'o  ~i 

wi,a^ 

*2  c  r:  £ 
c  q,(J  a 
X 


Tj- 

OO 


Table  F-l.  Model  Code  Release  Conditions,  Beginning  Effects,  Ending  Effects,  and  Decision  Nodes  (continued) 


LAR_RNG  I  ■  Dummy  3: 
absolute(p_brg_to  ;  Workload  Mgt 
_dp)>30 


This  page  left  blank  intentionally 


186 


APPENDIX  G 


DESCRIPTIONS  AND  CODE  ASSOCIATED  WITH  THE  USER-DEFINED 
MACROS  USED  IN  CASE  STUDY  1 
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Table  G-l.  Descriptions  and  Code  Associated  with  the  User-Defined  Macros  Used  in 

Case  Study  1 


BADJTHREATS 


Macro  Name 


Description 


Macro  Code 


SETJSTJINK 


INIT.ENTTYPE 


ICALC_DETECT 


:  In  the  scenario,  there  are  two  threats 
capable  of  reaching  the  aircraft  at  the 
tplanned  altitude.  This  macro  is  used 
i  lo  check  the  emitter  type  of  any 
launched  threat  and  to  determine 
whether  that  threat  is  one  of  the  two 
lethal  threat  types. 


?Once  the  evasive  heading  is 
determined  and  flown,  the  HPM  starts 
a  number  of  “jinks”  left  and  right 
l until  the  threat  is  dropped.  This 
macro  is  used  to  determine  whether 
fihe  next  jink  is  to  the  right  or  left  of 
the  evasive  heading. 


| This  macro  initializes  constants 
[associated  with  each  entity  type. 


This  macro  is  used  to  update  the 
moving  status,  gmti  hit  status,  and 
size  of  any  objects  in  the  sensor  field 
of  view.  This  information,  coupled 


j  |  determine  which  of  launched  threats  can  kill  you  at  altitude} 
?tht_index:=0; 

{ must_evade:=FALSE; 

Awhile  thtJndex<tht_count  do 
if  tht_emttyp[tht_index]  ==  K_EMT__TYPE_J  ! 
tht_emttyp[thMndex]  =  K_EMT_TYPE_2  then 
bad„tht[tht_index]:=TRUE, 
must_evade:=TRUE, 
tht_index+=l 
\  else 

bad_tht[tht_index]  :=FALSE, 
thtjndex+=l; 

■  tum:=int_min_hdg-p_heading; 
if  tum>=180  then 
'  tum:=tum-360; 
if  tum<(-180)  then 
tum:=tum+360; 
if  tum>=0  then 
nextJink:=K_JINK_LEFT; 
if  tum<0  then 

next Jink  :=K_J  IN  K_R  IGHT; 
l{ ini t  GROUND  MOVER  entity  type  constants} 

•  K_GND_M  1 A 1  :=200 1 ; 

;  K_GND_T72:=300 1 ; 

K_GN  D_TRUCK:=3002; 

K_GND_SCUD:=3003; 

;K_GND_T90:=3004; 
i  K_GN  D_T  APZ:=3005 ; 

K_GND_BRDM:=3006; 

;K„GND_BTR:=3007; 

K_GND_TRACKD:=3008; 

K_GND_VAN:=3009; 

K_GND_MTLB:=30 10; 

{ init  BUILDING  entity  type  constants } 

K_BLD_COM:=9001; 

K_BLD_MFG:=9002; 

K_BLD_PWR:=9003; 

K_BLD_BRIDGE:=9004; 

K_BLD_SAM:=9005; 

K_BLD_AIRFLD:=9006; 

i  { Input:  inputjljdx,  il_sensor 
Output:  obj_detected, 

.set  variables  to  examine  one  object  from  the  image  list  at  a  time} 
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Macro] 


Description  Macro  Code 


;  with  the  sensor  used,  is  used  to  evaI_type:=iLtype[input JUdx] ; 

isuSed^TSon  toe  sensor  <mg  is  converted  from  NM  to  ft  for  later  calculation} 

’display.  After  calling  another  macro  eval_mg:=(il_range[inputJUdx]*6080); 

;to  calculate  the  number  of  pixels  jfeva,  0  0  then 
required  to  detect  and  identify  the 

object,  it  returns  whether  the  object  is  eval_mg:=  999999; 

detected.  evaLmoving:=il_moving[input_JUdx]; 

:  eval_gmt_hit:=il_gmtLdet[inputJLidx]; 

.  evaLsize:=il_size[input  JlJdx] ; 

|  { set  angles  to  be  used  if  TIR  is  sensor  used } 
tan_angle:=eval_size/eval_mg; 
j  obj_angle:=arctangent(tan_angle); 

{calculate  pixels  displayed  for  any  given  range/sensor/FOV } 

1  if  il_sensor=K_WS AR.LOW  then 
px_displayed:=eval„size/80; 

\i  iI_sensor=K_WSAR_MED  then 
px_displayed:=eval_size/40; 
if  il_sensor=K_WSAR_LOW  then 
px_dispIayed:=evaLsize/20; 

|  if  il_senson=K_NSAR_LOW  then 
i  px_displayed:=eval_size/ 1 0; 
if  iLsensor^=K_NSAR_MED  then 
.  px_displayed:=eval_size/3; 
if  il_sensor=K_NSAR_HIGH  then 
px_displayed:=eval_size/l ; 

|  Jif  il_sensor=K_TIR_WIDE  then 

?  px„displayed:=obj_angle/0.0136; 

Jif  il_sensor=K_T[R_N  ARROW  then 
j  px_displayed:=obj_angle/0.0068; 

if  il_sensor=K_TIR_2X„NAR  then 
px_displayed:=obj_angle/0.001 9; 

{ for  DEBUG,  increase  px_displayed  by  some  factor} 
;px_displayed:=px_displayed  *  K_PX_FACTOR; 

{call  macro  to  determine  how  many  pixels  are  required  in  order  to 
[ detect/identify  each  object  type} 

CALC_PIX_REQ; 

■  { determine  whether  object  is  detected} 

I  if  eval_gmt_hit=TRUE  I  px_displayed>=px_to_detect  then 
obj_detected:=TRUE 


obj_detected:=FALSE; 


?CALC_IDENT  This  macro  is  used  to  update  the 

j  moving  status,  gmti  hit  status,  and 

i  size  of  anv  objects  in  the  sensor  field 


{ Input:  inpuUUdx,  il_sensor 
Output:  objjdent,  obj_is„toi 


Macro  Name  Description 


of  view.  This  information,  coupled 
Hvith  the  sensor  used,  is  used  to 
calculate  the  number  of  pixels 
subtended  by  the  object  on  the  sensor 
^display.  After  calling  another  macro 
lo  calculate  the  number  of  pixels 
^required  to  detect  and  identify  the 
i  object,  it  returns  whether  the  object  is 
‘  identified  and  whether  the  identified 
■object  is  of  the  same  type  as  the  target 
i  of  interest. 


Macro  Code  ______ 

set  variables  to  examine  one  object  from  the  image  list  at  a  time} 
cval_type:=iLtype[inputJLidx]; 

>  (mg  is  converted  from  NM  to  ft  for  later  calculation) 
evaLmg:=(il_range[inputJl_idx]*6080); 
if  eval_mg==0.0  then 
eval_mg:=  999999; 

t  evaLmoving:=iLmoving[inputJl  Jdx] ; 

-  cval_^mt_hit:=il^mti_det[input_il  Jdx] ; 

(evaI_size:=iLsize[input  JlJdx] ; 
j  { set  angles  to  be  used  if  TIR  is  sensor  used } 
l  nm_angIe:=evaLsize/e  val_mg; 
u>bj_angle;=arctangent(tan_angle); 

i  {calculate  pixels  displayed  for  any  given  range/sensor/FOV} 
jif  iLsensor=K_WSAR_LOW  then 
;  px_displayed:=evaLsize/80; 

[if  il_sensor=K_WSAR_MED  then 
:  px_displayed:=eval_size/40; 
iif  il_sensor=K_WSAR_LOW  then 
j  px_displayed:=eval_size/20; 

;if  il_sensor=K_NSAR_LOW  then 
px_displayed:=eval_size/10; 
if  iLsensor=K_NSAR_MED  then 
px_di  spl  ayed  :=eval_si  ze/3 ; 
if  il_sensor=K_NSAR_HIGH  then 
1  px_displayed:=eval__size/l ; 
iif  il_sensor=K_TIR__WIDE  then 
px_displayed:=obj_angle/0.0136; 
if  il_sensor==K_TFR_N  ARROW  then 
px_displayed:=obj_angle/0.0068; 

!  if  il_sensor=K_TIR_2X_NAR  then 
px_displayed:=obj„angle/0.0019; 

^  call  macro  to  determine  how  many  pixels  are  required  in  order  to 
detect/identify  each  object  type} 

C  ALC_PIX_REQ; 

{determine  whether  object  is  identified} 

obj_ident:=FALSE; 

k>bj_is_toi  :=FALSE; 

if  il_sensor<=K_NSAR_HIGH  &  eval_moving==TRUE  then 
;  obj_ident:=FALSE 


if  px_displayed>=px_to_ident  then 
obj_ident:=TRUE; 

{determine  if  identified  object  is  the  target  of  interest} 
iif  objJdent=TRUE  &  eval_type==K_GND_SCUD  then 
obj  _is_toi  :=TRUE; 


|CALC_NXTSNSR  i  his  macro  is  used  to  determine  the  (this  function  determines  the  next  sensor  to  be  used  for  items  detected  [ 
{  :next  desired  sensor  field  of  view  with  in  the  image,  | 


191 


Macro  Name  ;  Description  j  Macrocode 

which  to  view  a  detected  object  that  isj  jnput:  ii_sensor  (global), 

?in  TIR  range. 

output:  nxtsnsr} 

{if  ii_sensor<K_TIR_N ARROW  then 
nxtsnsr:=K_TIR_N  ARROW; 

j  :  if  i Lsensor>=K_TIR_N ARROW  then 

\  nxtsnsr:=K_TIR_2X_N  AR ;  _  _  _ 

CHOOSE_DEST  *This  macro  is  used  to  determine  when  ;  {This  macro  is  used  to  determine  when  trg_nav_nfnd  and 
I  the  planned  and  updated  weapon  trg_nav_abrt  are  set. 

jrelease  points  are  sequenced.  It  then  j,  rp_next_step,  p_acjat,  p_ac_lon,  K_ORIGWR_LAT, 
initiates  vanous  timers  used  to  tngger  «_OR1GWR_LON,  K_LATLON_ADJ,  upd_wrp_lat,  upd_wrp_lon, 
a  replan  or  mission  abort.  ctak,  wpt_tmpclock) 

outputs:  trg_nav_nfnd,  trg_nav_abrt,  give_up} 

;  {sense  when  a/c  is  at  original  WRP  on  1  st  pass} 

if  rp_next_step=l  &  absolute(p_ac_lat-K_ORIGWR_LAT[route])  < 
K_LATLON_  AD  J  &  absolute(p_acJon-KJDRIGWR_LON[route])  < 
K_LATLON_ADJ  then 

wpt_tmpclock:=clock, 
rp_next_step:=2; 

;  {if  need  a  second  pass,  wait  -30  sec  (.5  min)  to  make  sure  WRP  is 
sequenced,  then  trigger  nav.  }  I 

if  rp_next_step==2  &  (clock-wpt„tmpclock)>=0.5  & 
toi_found=FALSE  then 


trg_nav_nfnd:=TRUE, 
nav_nfnd_pdg:=TRUE, 
wpt_tmpclock:=  1 0000, 
rp_next_step:=3; 

{once  plan  to  refly  is  complete  (where  rp_next_step  is  set  to  4  and 
wpt_tmpclock  is  reset  to  clock)  make  sure  a  couple  of  minutes  have 
j  elapsed  before  we  begin  to  monitor  for  sequencing  WRP  on  second 
j  pass. } 

|  if  rp_next_step==4  &  (clock- wpt_tmpclock>=2.0)  then 
wpt_tmpclock:=  i  0000, 
rp_next_step:=5; 

if  rp_next_step=5  &  absolute(p_ac__lat- 
upd_wrp_lat)<K_LATLON_ADJ  &  absolute(p_ac_lon- 
upd_wrp_lon)<K_LATLON_ADJ  then 

wpt_tmpclock:=c!ock, 

rp_next_step:=6; 

;  {wait  3  min  to  make  sure  WRP  is  sequenced  &  out  of  TIR  mg,  then 
trigger  nav_abrt.  in  abort  tasks,  rp_next_step  will  be  set  to  99  such 
i that  this  macro  won’t  be  called  again } 

•  if  rp_next_step==6  &  (clock- wpt_tmpclock)>=3 .0  then 
trg_nav_abrt:=TRUE, 


give_up:=TRUE, 
wpt_tmpclock:=  1 0000, 
rp_next_step:=99; 


'The  detection  and  identification 
■  algorithms  in  the  model  rely  upon 
idata  derived  from  Johnson’s  Criteria. 
This  approach  states  detection  and 
identification  criteria  in  terms  of  the 
number  of  pixels  a  given  object 
subtends  on  a  display.  This  macro 


{input:  eval_type 

;  output:  pix_to_detect,  pix_to_ident} 
px_jo_detect:=1000; 
px_to_ident:=1000; 

{determine  pixels  needed  for  movers} 


192 


Description  Macro  Code 

assigns  a  number  of  pixels  required  to  ;  if  evaLtype==K_GND_M  1 A 1  then 

detect  and  identify  each  of  the  ground  ■ 

objects  thaf  exists  in  the  scenario.  ^ca*c  uses 

px_to_detect:=1.7596, 


|CALC_PIX_ 

{(continued) 


px_to_ident:=7.7064; 
if  eval„type==K_GND_T72  then 
{calc  uses. T-72} 
px_to_detect:=l  .7596, 
px_to_ident:=7.7064; 

! if  evaUype=K_GND_TRUCK  then 
:  {calc  uses  genjruck} 

I  px_to_detect:=2.I565, 

i  px_to_ident:=9.3391;  | 

I  if  evaLtype==K_GND_SCUD  then 
,  {calc  uses  TEL) 
px_to_detect:=  1 .411 8, 
px_to_ident:=6.2756; 

I  if  eval_type==K_GND_T90  then 
•  {calc  uses  T-90} 
px_to_detect:=l  .7537, 

:  px_to_ident:=7.6822; 
i  f  e  val_type==K_GND_T  APZ  then 
|  {calc  uses  fueMruck} 

;  px_to_detect  :=2 .2472, 

■  px_toJdent:=9.7122; 

:  if  eval_type==K_GND_BRDM  then 
{calc  uses  B RDM) 
px_to_detect:=2. 1 569, 
px_to_ident:=9.3409; 
if  evaLtype==K_GND_BTR  then 
|  {calc  uses  BTR-80} 

:  px_to_detect:=  1.7537, 

!  px_toJdent:=7.6822; 

[if  eval_type==K„GNDJTRACKD  then 
j  {calc  uses  BMP} 

I  px_to__detect:=  1.3029, 
px_toJdent:=5.8274; 
lif  eval_type==K_GND_VAN  then 
I  {calc  uses  gen_truck} 
px__to_detect:=2.1565, 
i  px„to_ident:=9.3409; 
if  eval_type==K_GND_MTLB  then 
{calc  uses  MTLBJJ} 
px_to_detect:=  1.8250, 

\  px_to_ident:=7.9753; 

{determine  pixels  needed  for  buildings,  all  calcs  use  cmd_post,  which 
representys  asymptote  of  det/ID  curve} 

Jif  eval_type==K_BLD_AIRFLD  then 
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Macro  Name 


Description 


Macro  Code 


CHK_ID_2_IMG 


CHKJDJSJMC 

(continued) 


px_to_detect:=l.3, 

px_to_ident:=5.4; 

if  eval_type==K_BLD_BREDGE  then 
px__to_detect:=l  .3, 
px_to_ident:=5.4; 
if  eval„type==K_BLD_COM  then 
px_to_detect:=l  .3, 
px_to_ident:=5.4; 
if  eval_type=K_BLD_MFG  then 
px_to_detect:-l  .3, 

!  px„to_ident:=5.4; 

:if  eval_type==K_BLD_PWR  then 

px_to_detect:=1.3, 

px_toJdent:=5.4; 

■  if  evaI_type==K_BLD_SAM  then 
px_to__detect:=1.3, 
px_toJdent:=5.4; 


{Update  the  PL  when  we  look  at  a  loc  at  which  an  object  is  not 
detectable. 


NOTE:  id__to  Jmage  is  set  to  - 1  for  ref  &  for  alternate  pt  looks. } 
found:=FALSE; 


This  macro  is  used  to  determine  the 
next  sensor  look  when  the  TIR  is 
placed  at  an  object’s  last  know 
location,  and  that  object  is  not 
^detected.  If  the  highest  resolution 
sensor  (TIR_2X_NAR)  was  used  and  found_status:=0; 
the  object  is  still  not  detected,  it  is  : 
assigned  for  shootlist  removal.  If  an  mdex:-(-l); 
object  is  detected  with  the  highest  if  id_to_image<>(- 1 )  then 
resolution  sensor  but  can’t  be  while  index  <  iar_num_obj-l  do 

identified,  it  is  classified  as 

“un  viable”  and  will  not  be  looked  at  index+=l , 

; again  until  a  given  amount  of  time  has 
passed.  (Actual  assignment  of  the 
unviable  flag  and  time  monitoring  is  found:=TRUE, 
performed  on  the  FRED  side.) 


if  iar_obj_id[index]  ==  id_to_image  then 


found_status:=iar_obj_stat[index]; 

;  { Update  Next_Sensor  in  OOI  Logic } 

!  if  id_to_image<>(- 1 )  &  found” FALSE  &  iar_num_obj<  1 00  & 

;il_sensor==K_TIR_N  ARROW  then 
{ Update  Next_Sensor  for  this  object  in  PL} 
iar_obj_id[iar_num_obj]  :=  id_to_image, 
iar_obj_nxts[iar_num_obj]  :=  K_TIR_2X_NAR, 
iar_obj_stat[iar_num_obj]:=K_UPD_SENSOR, 
iar_num_obj+=  1 ; 

{Remove  from  OOI  Logic} 

if  id_to_imageo(-l )  &  found==:FALSE  &  iar_num_obj<  1 00  & 
il_sensor==K_TLR„2X_NAR  then 

{remove  this  object  from  PL} 
iar_obj_id[iar_num_obj]  :=  id_to_image, 
iar_obj_stat[iar„num_obj]  :=K_REMO  V  E_PL, 
iar„num_obj+=I ; 

{ Logic  to  delay  looking  at  points  that  have  been  detected  but  not 

identified} 

index:=-l; 

I  l_num  Jar:=iar_num_obj: _ 
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Macro  Name 


Description 


IDETECT.OBJ 


|  This  macro  builds  the  initial  list  of 
;  “image  analysis  results”.  It  looks  at 
i  the  results  of  the  CALC_DETECT 
r macro  to  determine  whether  each 
object  in  the  image  was  detected. 


Macrocode 


while  index  <  Lnum_iar-1  do 
■;  index+=l, 

if  iar_num_obj<100  &  il_sensor==K_TIR_2X_NAR  & 

\ iar_obj_stat[index]==K_DETECTED  then 

{Start  Unviable  timer  for  this  object  in  PL} 
iar_obj_id[iar_num_obj]  :=  iar_obj_id[index], 
iar_obj_nxts[iar_num_obj]  :=  K_TIR_NARROW, 
iar_obj_stat[iar_num_obj]  :=K_UNVIABLE, 
iar_num_obj+=l ; 

'  { using  Image  List  from  FRED,  this  builds  a  list  (image  analysis 
results  or  ‘iar’)  of  objects  detected  ands  marks  them  as  'detected'} 

{ Changed  'count'  to  'ct'  to  eliminate  if  character  limit} 

fct:=0; 

Udx:=(-1); 

while  idx<il_number-l  do 


:  i f  iLpre v_id[idx]==FALSE&(i LsensoroK_R  EAL_B  E AM  I 
(iLsensor==K_REALBEAM  &  iLrng_ref[idx]<K__RBEAM_ROI)) 
ahen 

:  inputJLidx:=idx, 

CALC.DETECT, 
if  obj_detected==TRUE  then 
:  iar_obj_mvng[ct]  :=il_moving[idx], 

?  cns_moving:==iar_obj_mvng[ct], 

;  CALC_NXTSNSR, 

'  iar__obj_id[ct]  :=il_obj_id[idx], 
iar_obj_lat[ct]  :=il_lat[idx], 
iar_objjon[ct]:=iljon[idx], 
iar_obj_stat[ct]:=K_DETECTED, 

\  iar_obj_nxts[ct]:=nxtsnsr, 

;  detect_il[ct]:=idx. 


|CLR_ACQ_TRG 


CLR_ATK_TRG 


This  macro  is  used  at  the  end  of  the 
j  Acquisition  function  to  reset  the 
:  Acquisition  triggers. 

This  macro  is  used  at  the  end  of  the 
Attack  function  to  reset  the  Attack 


i  {store  number  of  entries  placed  in  the  table} 
iar_num_obj:=ct; 

{ store  the  number  that  were  detected--this  will  be  adjusted  for  any 
i objects  that  are  subsequently  identified} 

jiar_num_det:=ct; 
i  rrg_acq_rb:=FALSE; 
trg_acq_upd:=FALSE; 

itrg_attack:= FALSE; 


CLR  EVD  TRG 


This  macro  is  used  at  the  end  of  the 
i  Evade  function  to  reset  the  Evade 


\  trg_evade:=FALSE; 


CLR  NAV_TRG 


This  macro  is  called  at  the  end  of  the 
Navigate  goal  function  to  reset  the 


trg_  nav_abrt:=FALSE; 


trg_nav_pln:=FALSE; 
trg_nav_nfnd:= FALSE; 
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Macro  Name 


RANKJTHREATS 


Description 


•This  macro  is  used  to  rank  threats 
launched  simultaneously  against  the 
[aircraft.  It  bases  the  ranking  on  time 
5 to  intercept  (TTI)  which  is  passed  to 
(the  HPM  by  FRED.  While  TTI 
would  not  be  explicitly  available  to 
;the  pilot,  it  is  assumed  that  it  could  be 
■  estimated  based  on  the  time  and  range 
bf  the  launches  (info  that  is  available). 


Macrocode 


j  { If  there  are  multiple  launches  that  can  kill  you,  rank  them  based  on 
[time  to  intercept  to  find  the  most  dangerous  (top_threat) } 

thtjndex:=0; 

:shortest_tti:=  1000.0; 
while  thtJndex<tht_count  do 

.  if  bad_tht[thtJndex]=TRUE  & 
tht_msltti[thtJndex]<shortest_tti  then 

t  shortest_tti  :=tht_msltti[tht_index], 

i 

top_threat:=tht_index, 
curr_emitter:=tht_emtid[thMndex] , 
i  thtjndex+=l 


IDENTIFY.OBJ  [This  macro  builds  the  final  list  of 
“image  analysis  results”.  It  looks  at 
'the  results  of  the  CALC_IDENT 
I  macro  to  determine  whether  each 
[object  in  the  image  was  identified. 


tht_index+=l; 

;  {looks  at  image  list  from  FRED,  iar  data  from  Detect JDBJ,  detectjl, 
and  builds  a  list  of  objects  (iar)  that  were  ID'd.  It  also  identifies  if  the 
[TO I  has  been  found} 

[count:=0; 

Uoi_found:=FALSE; 

idx:=-l; 

[while  idx<iar_num_obj-l  do 


input_iLidx:=detect_ilIidx], 

CALC.IDENT, 
if  obj  _ident=TRUE  then 
iar_obj_stat[idx]:=K_IDENTIFIED, 
count+=l, 

if  obj_is_toi=TRUE  then 
toi_found:=TRUE, 
toi_index:=detect_iI[idx]; 

{code  does  not  yet  allow  mis-identification  of  objects} 
I  { store  the  number  of  objects  ID’d} 

\  iar_num_id:=count; 

{ reset  the  number  detected  based  on  number  id’d } 

1  iar_num_det:==iar_num_det-count; 

{set  sensor  used} 
i  iar_sensor:=il_sensor; 


REF_IN_RNG  This  macro  looks  at  the  range  and  { dete 

bearing  to  the  reference  to  determine  SAR 
whether  it  is  within  the  viewable  jn  , 
region  of  the  SAR  and/or  TIR. 


{determine  if  current  ref  point  is  in  mg  to  image  with  TIR  and/or 


In pu ts :  p_m g_to_ref,  p_brg_to_ref. 

[Outputs:  in_sar_mg,  in_tir_mg,  ref_in_mg} 

Hf  p_rng_to_ref<=K_SAR_RNG  &  absolute(p_brg_to_reO 
>=K„SAR_GIM_LO  &  absolute(p_brg_to_ref)  <= 
K_SAR_GIM_HI  then 

in_sar_mg:=TRUE 


i  n_sar_mg:= FALSE; 


196 


Macro  Name 


1NIT_ALT_ACQ 


JNIT.CMDJDS 


Description  Macro  Code 

if  p„mg_to_ref<=K_TIR_RNG  then 
[  in_jir_mg:=TRUE 
”e!se 

l  in_tir_mg:=FALSE; 

I  if  in_tir_mg=TRUE  I  in_sar_mg=TRUE  then 
ref_in_mg:=TRUE 
else 


ref_in_mg:=FALSE; 

iThis  macro  is  used  to  initialize  the  K_ALT_PT_LAT[1,1]:=46.3; 

! alternate  waypoints  for  each  mission.  ^  py  lqn[1  l]-=23  0' 
Mf  the  original  weapon  release  point  is 

i  sequenced  and  the  target  is  not  yet  i  K_ALT_PT_LAT[  1 ,2]:  =46. 2; 

\ found,  these  alternate  waypoints  will  <k  ALT  PT_LON[l,2]:=22.7; 
j  he  entered  into  the  planner  prior  to  ;  “ 

■: requesting  a  replan  to  refly  the  target  ;  K_ALT_PT_LAT[2, 1  ] .=45.7, 
Uirea.  ;  K_ALT_PT  JLON[2, 1  ):=23.4; 

i  K_ALT_PT_LAT[2,2]  :=45.9; 
K__ALT_PT_LON[2,2]  :=23 . 1 ; 
K_ALT_PT_LAT[3,1]:=45.6; 

;  K_ALT_PT_LON[3, 1  ]  :=22.3; 

:  K_ALT_PT_LAT[3,2]  :=45.5; 

1  K„ALT_PT_LON[3,2]  :=22.7; 

\  j  K_ALTJPTJLAT[4, 1  ]  :=46.4; 

j  K_ALT_PT_LON[4, 1  ]  :=22.8; 

1  K_ALT_PT_LAT[4,2]  :=46.2; 
K_ALT_PT_LON[4,2]  :=22.6; 

:  K_ALT_PT_LAT[5, 1  ]  :=45.57; 
:  K_ALT_PT_LON[5, 1  ]  :=22.07; 
;  K_ALT_PT_LAT[5,2]  :=45.7 1 ; 
K_ALT_PT_LON[5,2]  :=23.00; 
K_  ALT_PT_LAT[6, 1  ]  :=46.4; 
K_ALT_PT_LON[6, 1  ]  :=22.45; 


‘  K_ALT_PT_LAT[6,2]  :=46.35; 

;  K_ALT_PT_LON[6,2]:=22.8; 

:  (initialize  the  number  of  alt  waypoints  for  each  of  the  six  routes} 


|  K_NUM_ALT_PT[  1  ] 
K_NUM_ALT_PT[2] 
: K^NUM_ALT_PT[3] 
::  K_NUM_ALT_PT[4] 


: K_NUM_ALT_PT[5]:=2; 
K_NUM„ALT_PT[6]:=2; 

This  macro  specifies  the  enumeration  K_REL_WCM:=1; 
for  each  of  the  HPM  commands  given  ^  gaMEOVER'=15- 
io  FRED.  " 

)  K_U  FC_PLNMOD  :=2; 


K_UFC_TGTM0D:=3; 
?K_D0I_LFTMPD:=1 1; 
K_D0I_RTMPD:=12; 
K_DOLCTRMPD:=10; 
K_CURS0R_AC:=8; 
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Macro  Name 


Description 


Macro  Code 


INIT_GLOBAL 


IThis  macro  initializes  all  of  the 
[scenario-independent  variables  prior 
to  the  start  of  a  trial. 


1NIT_GL0BAL 

(continued) 


K_DESIG_PT  :=  1 3 ; 

K_M  AK_MSTFLY  :=  1 4; 

K_MOVECURSOR;=9; 

K_F_ACP_RPLN:=6; 

K_REQ„RPLN:=4; 

l  K_I_LACP_RPLN  :=7 ; 

K  JR  EQ_  ABORT  :=5 ; 

j  K_DEPLOY  JTIR:=24; 

I  K_STOW_TIR:=23; 

i  K_A_PILOT_D:=2 1 ; 

\  K_A_PILOT_E:=20; 

iK_RESETLSL:=22; 

i  K_REL_CMEAS:=  1 6; 

IK  EVADE  MNVR:=25; 
,  —  — 

!K_THROTTLE:=19; 

K_A_THROTTLE:=l  8; 

K_RM  V_SL_OB  J  :=  1 7 ; 

t  K_U  FC_TGT_LL : =27 ; 

\  KJJFC_LL_ACP:=26; 

;  K  J(JPD_REFLOC:=29; 

KJRECALC_PL:=33; 

K__RESET_OOI:=32; 

K_CALC_ALTPT  :=34; 

K_GEN_IM  AGE:=28: 

K„PROCES_OOI:=30; 

i  K_RANK_OOI:=3 1 ; 

;  {set  other  constants} 

K_DETECTED:=  1 ; 

\  K_IDENTIFIED:=2; 

K_REMOVE_PL:=3 ; 

KJJPD_SENSOR:=4; 

K_UNVIABLE:=5; 

K_EMT_TYPE_1  :=7405; 

;K_EMT_TYPE^2;=7015; 

K_TIR_2X_NAR:=9; 

K_TIR_NARROW:=8; 

i  KJTIR_WIDE:=7; 

\  K_NSAR_HIGH:=6; 

K_NSAR_MED:=5; 

K_NSAR_LOW:=4; 

!  K_WSAR_HIGH:=3; 

K_WSAR_MED:=2; 

i  K_WS  AR  JLOW  :=  1 ; 

K_REAL_BEAM:-0; 

K_DELAY_END:=1; 

K_LAR_RNG:=6; 

K_HOM  E_LAT  :=43 .50; 
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Macro  Name 


Description 


Macro  Code 


!NIT_GLOBAL 

(continued) 


K_HOME_LON:=  1 6.299999 17; 

K_S  AR_RNG:=70; 

K_TIR_RNG:=20; 
j  K_SAR_GIM_HI:=60; 
K_SAR_GIM_LO:=0; 
fK_DFLTjSNSR:=K_WSARJLOW; 
j  K_BINGO_FUEL:=2000; 

\  K_RP_TYP_REQ:=0; 
K_RP_TYP_NAV:=2; 
r  K_RP_TYP_THT:=3 ; 
;K_RP_TYP_FUL:=4; 
!K_RP_TYP_ERR:=5; 
K_LATLON_ADJ:=.04; 
K_DOLLEFT:=l; 
j  K_DOI_C  ENTER  :=2; 
KJX)IJRIGHT:==3; 
j  K_U  FC_PLAN  := 1 ; 

K_UFC_NAV:=2; 

•K_UFC_TGT:=3; 

K_HDG_ADJ:=5.0; 

K„JINK_LEFT:=1; 

;KJINK_RIGHT:=2; 

K_M  AX_ALT_LK:=5 ; 
j  K_PX_FACTOR:=  1 ; 

!  K_RBEAM_ROI:=  1 0; 

|TRUE:=1; 

iFALSE:=0; 

ON:=l ; 

K)FF:=0; 

^AR:=0; 

TrR:=0; 

|CRUISE:=1; 

FULL_MIL:=2; 
i  AFTERBURNER  :=3 ; 
j  AUTOTHROTTLE:=4; 
j  { initialize  variables  for  start  of  mission } 
\  hpm_cmd  Jd:=K_PROCES_OOI; 
fhpm_rti_trig+=l; 

|current_doi:=2; 

auto_pilot:=ON; 

1  must__evade:=FALSE; 

}curjhrottle:=CRUISE; 

tgt_update:=FALSE; 

pl_number:=0; 

index_to_alt:=0; 

•pass:=!; 

started  mer:=  10000; 
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Macro  Name 


Description 


|lNIT_LK_PTl 


This  macro  initializes  the  alternate 
lookpoints  around  the  original  and 
il updated  reference  points  for  Scenario 
:  l .  These  points  were  created  a  prior 
i  using  a  map  of  the  gaming  area.  They 
^represent  points  that  lay  on  road 
j networks  surrounding  both  the 
l  planned  and  updated  reference  points. 


Macro  Code 


]next_jink:=0; 
i  need__reflook:=TRUE; 
jlastpLcount:=(-l); 

|  ufc_mode:=K_UFC_PLAN ; 
pre_tir_upd:=FALSE; 

I  pre_tir_pndg:=FALSE; 

lrp_next_step:=l; 

ialt_pt_index:=l; 

i  wpt_tmpclock:=  1 0000;  _  _  _ _ _ _ _ _ 

{ Define  the  look  points  around  the  initial  and  updated  target 
i  coordinates  for  Route  i .  Uses  3D  array,  [route  number,  initial  (1)  v. 
updated  (2),  look  number  (0-4)} 

I  K_LK_PT_LAT[  1,1,0]  :=  45.86670; 
f  i  K_LK_PT_LAT[  1 , 1 , 1  ]  :=  45.84833 ; 

!  K_LK_PT_LAT[  1,1,2]  :=  45.852 1 7 ; 

Ik_LK_PT_LAT[1,I,3]:=  45.83000; 
l  K_LK_PT_LAT[  1 , 1 ,4]  :=  45.82583 ; 

|;K^LK_PT_LON[1,1,0):=  22.90000; 

}K_LK_PT_LON[  1,1 , 1  ];=  22.90733; 

[KJLK_PT_LON[l,l,2];=  22.904083; 

Ik_LK_PT_LON[1, 1,3]:=  22.94633; 

!  K_LK_PT_LON  [1,1,4]:=  22.91800; 

K_LK_PT_LAT[1 ,2,0]:=  45.85770; 
i  K_LK_PT_LAT[  1,2, 1]:=  45.85833; 

.  K_LK_PT_LAT[  1 ,2,2]  :=  45 .84 1 67 ; 
j  K_LK_PT_LAT[  1.2,3]:=  45.83083; 

|  K_LK_PT_LAT[  1 ,2,4]:=  45.83500; 

:  K_LK_PT_LON[  1 ,2,0]  :=  22.9 1490; 

:K_LK_PT_LON[l,2,l]:=  22.94333; 

|  K_LK_PT_LON[  1 ,2,2]:=  22.89500; 

\  K_LK_PT_LON[  1,2,3]:=  22.92083; 
iK_LK.PT_LON[  1,2,4]:=  22.95167; 


UNIT  LK.PT2 


This  macro  initializes  the  alternate 
lookpoints  around  the  original  and 
updated  reference  points  for  Scenario 
-2.  These  points  were  created  a  prior 
using  a  map  of  the  gaming  area.  They 
represent  points  that  lay  on  road 
networks  surrounding  both  the 
planned  and  updated  reference  points. 


{ Define  the  alternate  look  points  around  the  initial  and  updated  l 
coordinates  for  Route  2.  Uses  3D  array,  [route  number,  initial  ( 
updated  (2),  look_pt  (04) } 

I  K_LK_PT_LAT[2, 1 ,0]:=  45.90000; 

SK_LK_PT_LAT[2, 1 ,  l]:=  45.91550; 

;  K_LK_PT_LAT[2, 1 ,2]  :=  45.90500; 

K_LK_PT_LAT[2, 1,3]:=  45.91500; 

( K_LK_PT_LAT[2, 1 ,4]:=  45.9 1 167; 

K_LK_PT_LON[2, 1 ,0]:=  22.75000; 

K_LK_PT_LON[2, 1,1]:=  22.73333; 

K_LK_PT_LON  [2,1,2]:=  22.73700; 

K_LK_PT_LON[2,l,3J:=  22.76167; 
i  K_LK_PT_LON[2,l  .4]:=  22.78 167; 

:  K_LK_PT_LAT[2,2.0]:=  45.89 1 80;  _ _ 
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Macro  Name  Description  Macro  Code 


?  K_LK_PT_LAT[2,2, 1  ]:=  45.90000 
j  K_LK_PT_LAT[2,2,2] :=  45.91000 
!K_LK_PT_LAT[2,2,3]:=  45.91500 
S  K_LK_PT_LAT[2,2,4]  :=  45.91 167 
j  K_LK_PT_LON  [2,2,0]  :=  22.74610 
;  K„LK_PT_LON[2,2, 1  ]  :=  22.73800 


I  K__LK_PT_LON[2,2,2]  :=  22.73800: 
i  K_LK_PT_LON  [2,2.3]  :=  22.76 1 67 
j  K_LK_PT_LON  [2,2,4]  :=  22.78 1 67 

INIT_LK_PT3  [  This  macro  initializes  the  alternate  f  { Define  the  alternate  look  points  around  the  initial  and  updated  target 

lookpoints  around  the  original  and  coordinates  for  Route  3.  Uses  3D  array,  [route  number,  initial  (1)  v. 

^updated  reference  points  for  Scenario  ^updated  (2),  look_pt  (0-4) } 

\ X  These  Poinls  were  created  a  Pnor  K_LK_PT_LAT[3, 1 ,0]  :=  45.88333; 
musing  a  map  of  the  gaining  area.  They 

represent  points  that  lay  on  road  |K_LK_PT_LAT[3,1,1]:=  45.89500; 

;  networks  surrounding  both  the  K_LK_PT_LAT[3,1,2]:=  45.90417; 

;  planned  and  updated  reference  points. 

|  i  K_LK_PT_LAT[3, 1,3]:=  45.90667 ; 

K_LK_PT_LAT[3, 1 ,4]  :=  45 .9 1 000: 


K_LK_PT_LON[3,  l  ,0]  :=  22.88333 
i  K_LK_PT_LON[3, 1,1]:=  22.89833 
K_LK_PT_LON[3, 1 ,2]  :=  22.88167 
K_LK_PT_LON[3, 1,3]:=  22.87500 
j  K^.LK_PT_LON[3,l,4]:=  22.87500 
j K_LK_PT_LAT[3,2,0]  :=  45.89520 
K_LK_PTJLAT[3,2, 1  ]  :=  45.88333 
jK_LK_PTLLAT[3,2,2]:=  45.90833 
K_LK_PT_LAT[3,2,3]  :=  45.91000; 


:K„LK_PT_LAT[3,2,4]:=  45.89333; 
K_LK_PT_LON[3,2,0]  :=  22.88200: 
j  K„LKJPT_LON[3,2, 1  ]  :=  22.87667: 


:K_LK_PT_LON[3.2,2]:=  22.87333 
lK_LK_PT_LON[3,2.3]:=  22.90333 
jK_LK_PT_LON  [3,2,4]:=  22.90500 


IN1T_LK_PT4 


This  macro  initializes  the  alternate  ■  { Define  the  alternate  look  points  around  the  initial  and  updated  target 
lookpoints  around  the  original  and  coordinates  for  Route  4.  Uses  3D  array,  [route  number,  initial  ( 1)  v. 

updated  reference  points  for  Scenario  ^updated  (2),  Iook_pt  (0-4)} 

4.  These  points  were  created  a  prior  I  K_LK_PT_LAT[4, 1,0] :=  45.91660; 
fusing  a  map  of  the  gaming  area.  They 

; represent  points  that  lay  on  road  ;  K_LK_PT_LAT[4,1 , 1]:=  45.92217; 

i networks  surrounding  both  the  \  k_LK_PT_LAT[4,  1 ,2]  :=  45.93333 ; 

■  planned  and  updated  reference  points. : 

)  K_LK_PT_LAT[4, 1 ,3)  :=  45.94850; 

::  K_LK_PT_LAT[4, 1 ,4]  :=  45.94500: 


;  K_LK_PT_LON[4, 1 ,0]  :=  22.76660 
K_LK_PT_LON  [4,1,1]:=  22.76867 
K_LK_PT_LON  [4, 1 ,2]  :=  22.75167 


j  K_LK_PT_LON[4,  l  ,3]  :=  22.764 1 7 
i  K_LK_PT_LON[4, 1,4]:=  22.78667 
i  K_LK_PT_LAT[4,2,0]  :=  45.94490 
K_LK_PT_LAT[4,2, 1  ]:=  45.94800 
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Macro  Name 


INIT_LK_PT5 


'Description  :  |  --  '  :  :  Macro  Code  ■■  -  ~ 


i  K_LK_PT_LAT[4,2,2]:=  45.95167; 

|  K_LK_PT_LAT[4,2,3]  :=  45.93500; 

I  I  K_LK_PT_LAT[4,2,4]  :=  45.93333; 

\  i  KJLK_PT_LON[4,2,0]  :=  22.79290; 

\  )  K_LK_PT_LON[4,2, 1  ]:=  22.77500; 

|  K_LK_PT_LON[4,2,2]  :=  22.76000; 
j  K_LK_PT_LON[4,2,3]:=  22.75000; 

:K_LK_PX.LON[4,2,4]:=  22.78 167;  _ 

ithis  macro  initializes  the  alternate  :  { Define  the  alternate  look  points  around  the  initial  and  updated  target 
lookpoints  around  the  original  and  coordinates  for  Route  5.  Uses  3D  array,  [route  number,  initial  (1)  v. 
updated  reference  points  for  Scenario  updated  (2),  look_pt  (0-4)} 

5.  These  points  were  created  a  prior  ]k_LK_PT_LAT[5,  1 ,0] :=  45.88330; 
fusing  a  map  of  the  gaming  area.  They!  _ 

•  represent  points  that  lay  on  road  K_LK_PT_LAT[5, 1,1]:=  45.90000, 

networks  surrounding  both  the  iK_LK_PT_LAT[5,l,2]:=  45.86583; 

planned  and  updated  reference  points,  45  84433; 

iK_LK_PT_LAT[5, 1 ,4]:=  45.84 1 67; 

;K_LK_PT_LON[5,1,0]:=  22.80000; 
j  K_LK_PT_LON[5,l,l]:=  22.80500; 

•  f:K_LK_PT_LON[5,l,2]:=  22.79833; 

I  }K_LK_PT_LON[5, 1,3]:=  22.81500; 

I  1  K_LK_PT_LON[5, 1 ,4]  :=  22.83500; 

|  !  K_LK_PT_LAT[5,2,0]  :=  45.85 1 60; 

j  iK_LK_PT_LAT[5,2,l]:=  45.85833; 

|  1  K_LK_PT_LAT(5,2,2]  :=  45.85333; 

K_LK_PT_LAT[5, 2,3):=  45.84333; 

K_LK_PT_LAT[5, 2,4]:=  45.84750; 

!  K_LK_PT_LON[5,2,0]  :=  22.79 1 1 0; 

K_LK_PT_LON[5 ,2,1]:=  22.79333; 
l  jK_LK_PT_LON[5,2,2]:=  22.80000; 

K_LK_PT_LON[5,2,3]  :=  22.82667; 
j  K_LK_PT_LON[5,2,4]:=  22.80333; 


INIT_LK_PT6  This  macro  initializes  the  alternate 

,  lookpoints  around  the  original  and 
updated  reference  points  for  Scenario 
i6.  These  points  were  created  a  prior 
‘using  a  map  of  the  gaming  area.  They 
represent  points  that  lay  on  road 
networks  surrounding  both  the 
planned  and  updated  reference  points, 


>  {Define  the  alternate  look  points  around  the  initial  and  updated  target 
coordinates  for  Route  6.  Uses  3D  array,  [route  number,  initial  (1)  v. 
updated  (2),  Iook_pt  (0-4)} 
i  K_LK_PT_LAT[6, 1 ,0]:=  45.95000; 
i  K_LK_PT_LAT[6, 1,1]:=  45.93500; 

I  K_LK_PT_LAT[6, 1 ,2]  :=  45.94500; 
i  K_LK_PT_LAT[6, 1 ,3]  :=  45.96450; 
i  K_LK_PT_LAT[6, 1 ,4]  :=  45.97833; 

|  K_LK_PT_LON[6, 1 ,0]  :=  22.76670; 

K_LK_PT_LON[6, 1,1):=  22.75500; 

}  K_LK_PT_LON[6, 1,2]:=  22.7891 7; 

:  K_LK_PT_LON[6, 1 ,3]:=  22.78800; 

:  K_LK_PT_LON[6, 1 ,4]  :=  22.80333; 

K_LK_PT_LAT[6,2,0]  :=  45.95780; 

K_LK_PT_LAT[6,2, 1  ]:=  45.97000; 

K_LK_PT_LAT[6,2,2]  :=  45.97833; 


Macro  Name 


Description 


Macro  Code 


INIT.SARJLKQ 


The  model  was  built  on  the 
assumption  that  the  SAR  leg  of  the 
acquisition  phase  would  be  used  only 
for  object  detection  and  building  up  a 
;  shootlist.  This  macro  defines  the 
•sequecence  of  SAR  images  to  be 
;  taken  during  that  leg.  It  consists  of  16 
-  images  that  include  the  reference 
\ point  and  four  alternate  lookpoints  at 
i  multiple  resolutions. 


|K_LK_PTJLAT[6,2,3]  :=  45.98167; 
jK_LK_PT_LAT[6,2,4]  :=  45.97000; 

!  K_LK_PT_LON  [6,2,0]  :=  22.78150; 
j  K_LK_PT_LON[6,2, 1  ]  :=  22.79667; 

1  K_LK_PT_LON[6,2,2]  :=  22.80333; 
j  K_LK_PT_LON[6,2,3]  :=  22.78833; 
j  K_LK_PT_LON  [6,2,4]:=  22.76833: 
j{ Initialize  the  SAR  Look  Points  Queue) 

{SAR  Look  Points  -  Use  same  Lat/Long  index  for  Initial  &  Update) 
K_SJLP_MAX:=16; 

{For  *IDX,  0=>Ref  Pt  Coord,  l-4=>Alt  Pt  1-4  Coord) 


1NIT_SAR_LKQ 

[(continued) 


tlNIT  SPECIFC 


This  macro  initializes  the  lat/lon  of 


K_S_LP_EDX[0]  :=0; 

K_S_LP_IDX  [  1  ] : =0; 

K_S_LP„IDX[2]  :=0; 

K_S_LP_IDX[3]:=1 
K_SJ_P_IDX[4]:=1: 

K_S_LP_EDX[5]:=2: 

K_S_LP_IDX[6]  :=2 
K_S_LP_IDX[7]:=3: 

K_S_LP_EDX[8]  :=3: 

K_S_LP_IDX[9]  :=4: 

K„S_LP_EDX[  1 0]  :=4; 

|K„S_LP_IDX[1 1]:=0; 
jlC_S_LPJDX[12]:=l; 
j  K_S_LP_  IDX  [  1 3  ]  :=2 ; 
j  K_S„LP_IDX[  1 4]  :=3 ; 
j  K_S  JLP_IDX[  1 5]  :=4; 

!  K_S_LP_SNSR[0]  :=K_REAL_B  EAM ; 

|k_S_LP_SNSR[  1  ]  :=K_WS  AR_M  ED; 

lK_SJLP_SNSR[2]:=K_NSARJLOW; 

iK_S_LP_SNSR[3]:=K_WSAR_MED; 

jK_S_LP_SNSR[4]:=K_NSAR_LOW; 

|k_S_LP_SNSR[5]:=K_WSAR_MED; 

|K_S_LP_SNSR[6]:=K_NSAR_LOW; 

|K_S_LP_SNSR[7]:=K„WSAR_MED; 

IK_S_LP_SNSR[8]:=K_NSAR_LOW; 

K_S_LP_SNSR[9]  :=K_WSAR_MED; 

■  K_S_LP_SNSR[  1 0]  :=K_NS  AR_LOW ; 
;K_S_LP_SNSR[1 1]:=K_REAL_BEAM; 

1  K_S_LP_SNSR[  1 2]  :=K„WSAR_MED; 
j  K_S_LP_SNSR[  1 3]  :=K_WS  AR_MED; 

I  K_S_LP_SNSR[1 4]  :=K_WSAR_M  ED; 
i  K_S_LP_SNSR[  1 5]  :=K„WS  AR_M  ED; 
:next_isarjp:=0; 

!next_usar_lp:=0; 

1  {define  lat  Ion  of  planned  weapon  release  point) 


the  original  weapon  release  point  and  iK_ORIGWR_LAT[l  J:=45.86666472; 
the  “known”  lat/lon  of  the  target  of 


203 


interest  at  the  time  of  mission  1k_ORIGWR_LON[  1  ]:=22.89999< 

jp'amiHg.  This  lat/lon  value  is  used  to;  "  LAT[1]:=45.689; 

? set  the  initial  hpnurefjat  and  j'  - 

hpm_reMon  variables.  The  lat/iongs  Sk_ORIGWR_LON[1]:=22.533;} 
include  scenario  specific  values  for  qrIGWR  LAT[2]:=45. 900001 
each  of  six  scenarios.  i  “ 


Macro  Name  Description  .  ■  "j  \  ■  '  Macro  Code 


jK_ORIGWR_LON[  1  ]:=22.89999944; 
i  { K_ORIGWR_LAT[  1  ]  :=45.689; 


INIT_SPECIFC 

(continued) 


tINIT_TlR_LKQ 


Sensor  employment  during  the  TIR 
leg  of  the  acquisition  cycle  is 
intended  primarily  for  target 
identification.  The  model  will 
attempt  to  use  the  TIR  to  examine  and 


jK_ORIGWR_LAT[2]:=45. 90000 139; 
|k_ORIGWRJLON[2]:=22.75; 

K_ORIGWR_LAT[3]:=45.883335; 
K_ORIGWRJLON(3j:=22.88333306; 
j  K_ORIGWRJLAT[4]  :=45.9 1 66; 

|K_ORIGWR_LON[4]:=22.7666; 

!{[5]  was  45.87284833,  22.6121805) 

|  K_ORIGWR_LAT[5]  :=45.8544; 

|  K_ORIGWR_LON[5]  :=22.724 1 ; 

|  K_ORIGWR  _LAT[6]  :=45.9 1619861 ; 

I  K_ORIGWR_LON[6]  :=22.82497 ; 

{define  lat  Ion  of  waypoint  that  follows  the  planned  weapon  release 
point  -  COMMENTED  OUT!) 

{ K_POSTWR_LAT[  1  ]  :=43.5 ; 

K_POSTWR_LON[  1  ]  := 1 6.299999 17; 

K_POSTWR_LAT[2]:=45 .95697; 

K_POSTWR_LON[2]  :=2 1 .20853972; 
K_POSTWR_LAT[3]:=43.5; 

K_POSTWR_LON[3]  :=16.299999 17; 

K_POSTWR_LAT[4]  :=45. 62026972; 
j  K_POSTWR_LON  [4]  :=2 1 .9 1 443806; 
i  K_POSTWR_LAT[5]  :=45 .7789686 1 ; 
j  K_POSTWR_LON[5]  :=22.59836944; 
j  K_POSTWR_LAT[6]  :=43 .5 ; 

|  K_POSTWR_LON[6]  :=  16.299999 17;} 

j  {define  lat  Ion  of  target  as  known  at  the  time  of  mission  planning) 
j  K_TGTBEG_LAT[  1  ]  :=45.8667; 
j  K_TGTBEG_LON[  1  ]  :=22.9; 

|  K_TGTBEG_LAT[2]  :=45.9; 
j  K_TGTBEG_LON[2]  :=22.75; 

I  K_TGTBEG_LAT[3]  ;=45.8833; 

|  K_TGTBEG_LON[3]  :=22.8833 ; 

;  K  JTGTBEG_LAT[4]  :=45.9 1 66; 

!K_TGTBEG_LON[4]  :=22.7666; 

!  K_TGTBEG_LAT[5]  ;=45.8833 ; 

jKJTGTBEG_LON[5]:=22.8; 

lK_TGTBEG_LAT(6]:=45.9333; 

K_TGTBEG_LON[6]  :=22.75; 

{set  initial  reference  point  based  on  route  (from  FRED)) 
hpm_ref_lat:=K_TGTBEG_LAT[route] ; 

•  hpm_ref_lon  :=K_TGTB  EG_LON  [route] ; 

{Initialize  the  TIR  Look  Points  Queue) 

,  {TIR  Look  Points  -  Use  same  Lat/Long  index  for  Initial  &  Update) 
K  T_LP_MAX:=10; 
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Macro  Name 


Description 


Macro  Code 


SET  SAR_LKPT 


identify  anything  that  was  detected 
^and  added  to  the  shootlist  during  the 
SAR  leg.  If  however,  there  are  no 
viable  objects  on  the  shootlist  at  the 
time  the  aircraft  is  in  TIR  range  to  the 
reference  point,  the  TIR  will  be  used 
to  examine  the  reference  point  and  the 
|\four  altemate.look  points.  This  macro 
specifies  the  sequence  of  these  TIR 
looks. 


{For  *IDX,  0=>Ref  Pt  Coord,  l-4=>Alt  Pt  1-4  Coord) 
I  K_T_LP_IDX[0]  :=0; 

:  K_T_LP_IDX[  1  ]  :=0; 

\  K _T_LP_IDX  [2]  :=  1 ; 

K_T_LPJDX[3]:=1; 

K_T_LP_IDX[4]  :=2; 
j  K„T_LP_IDX[5]:=2; 

|KJT_LPJDX[6j:=3; 

(KJTJLPJDX[7]:=3; 
j  K_T_LP_IDX[8]  :=4; 
j  K_T_LP_IDX[9]  :=4; 
j  K_T_LP_SNSR  [0]  :=K_TIR_N  ARROW; 
K_TJLP_SNSR[  1  ]  :=K  _TIR_2X_N  AR ; 

§  KJL.LP_SNSR[2]  :=K_TER^N  ARROW; 

I  K_.T_LPL.SNSR  [3]  :=K_TER_2X_NAR ; 
K_T_LP_SNSR[4]:=K_TIR_NARROW; 
K_T_LP_SNSR[5]:=K_TIR_2X_NAR; 
j  K_T_LP_SNSR[6]  :=K_TIR_N  ARROW; 
j  K_T_LP_SNSR  [7]  :=K_TIR_2X_NAR; 
j  K_T_LP_SNSR  [8]  :=K_TIR_N  ARROW; 
i  K_T_LP_S  N  S  R  [9]  :=K  JTIR_2X_NAR; 
inextjtirjp:=0; 

5 

}next_utir_lp:=0; 

r 


'Phis  macro  is  used  to  set  a  SAR  look 
pointer  to  either  the  original  or 
updated  reference  and  alternate 
lookpoints,  based  on  whether  the 
target  update  has  been  completed.  In 
addition  it  sets  a  flag  if  the  current 
look  point  is  an  alternate  lookpoint. 
(This  has  implications  for  display  of 
interest  selection  when  imaging). 


if  tgt_upd_done==TRUE  then 
{Use  Update  Queue) 
idx:=K_S_LP_IDX[next_usar_lp], 

{ idx  now  contains  pointer  to  lat/long  of  ref/alt  look  pts } 
;  { Note:  The  T  indicates  use  of  the  update  coord } 
lat_to_i  mage  := K_LK  JPT_LAT[route,2,idx], 
lon_to_image:=K_LK_PT_LON[route,2,idx] , 

:  sensor_2_use:=  K_S_LP_SNSR[next_usarJp], 

:  next_usar_lp+=l 
else 

I  {Use  Initial  Queue) 
idx:=K_S_LP_IDX[next_isarJp], 

{ idx  now  contains  pointer  to  lat/long  of  ref/alt  look  pts } 
!  { Note:  The  ’1  ’  indicates  use  of  the  initial  coord ) 
lat„to  Jmage:=K_LK_PT_LAT[route,  1  ,idx), 
lon_to_i  mage: =K„LK_PT_LON[  route,  1  ,idx], 

•  sensor_2_use:=  K_S_LP_SNSR[next_isarJp], 
next  Jsar_lp+=  1 ; 

{Set  use  of  Alt  Pt  or  not) 

'if  idx==0  then 
alt_pt_inuse:= FALSE 
:clse 

alt_pt_inuse:=TRUE; 
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SET_TIR_LKPT  This  macro  is  used  to  set  a  TIR  look 
pointer  to  either  the  original  or 
updated  reference  and  alternate 
lookpoints,  based  on  whether  the 
target  update  has  been  completed.  In 
•  addition  it  sets  a  flag  if  the  current 
look  point  is  an  alternate  lookpoint. 
^(This  has  implications  for  display  of 
interest  selection  when  imaging). 


I 


SET_TIR_LKPT 

(continued) 


next_isarjp:=l  1; 
if  tgt_upd_done=TRUE  then 
{Use  Update  Queue} 
idx:==K_TJLPJDX[next_utirJpj, 

(idx  now  contains  pointer  to  lat/long  of  ref/alt  look  pts} 
{Note:  The  'T indicates  use  of  the  update  coord) 
lat_to_image:=K_LK_PT_LAT[route,2,idx], 
lon_toJmage:=K_LK_PT_LON[route,2,idx] , 
sensor_2_use:=  K_T_LP_SNSR[next_utirJp], 
next_utirjp+=l 
else 

{Use  Initial  Queue) 

idx  :=K_.TJ-PJDX  [next JtirJp], 

{ idx  now  contains  pointer  to  lat/long  of  ref/alt  look  pts } 
{ Note:  The  *1  ’  indicates  use  of  the  initial  coord } 
lat_to_image:=K_LK_PT_LAT[route,  l,idx], 
lon_to_image:=K„LK_PT„LON  [route,  1  ,idx], 
sensor_2_u  se  :=  K_T_LP_SNSR  [next  Jtir Jp] , 
nextjtirjp+=l; 

:  { Set  use  of  Alt  Pt  or  not) 
if  idx=0  then 
alt_pt_inuse:=FALSE 
else 

alt_pUnuse:=TRUE; 

{ Handle  wraparound) 
if  next_utir.jp  >=  K_T_LP_MAX  then 
next_utirjp:=0; 

if  nextjtirjp  >=  K_T_LP_MAX  then 
nextjtirjp:=0;  _ 


Macro  Name 


Description 


Macro  Code 


i  { Handle  wraparound } 
lif  next_usarjp  >=  K_S_LP_MAX  then 
i  next_usarjp:=l  1; 
if  nextjsarjp  >=  K_S_LP_MAX  then 
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APPENDIX  H 


SAMPLE  CALCULATION  OF  MEAN  RANKS 


207 


This  page  left  blank  intentionally 


208 


Table  H-l.  Sample  Calculation  of  Mean  Ranks 
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Table  1-1.  Average  Mean  Ranks 
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Table  1-1.  Average  Mean  Ranks  (continued) 


Maximum  61.  (XXX)  64.(XXX)  76.0(XX)  77.0000  58.0000  44.5000  56.5000  63.0000 


Table  1-1.  Average  Mean  Ranks  (continued) 
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APPENDIX  J 


CORRELATIONS  AMONG  THE  DEPENDENT  MEASURES 
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Table  J-l.  Correlations  Among  the  Dependent  Measures  for  the  Entire  Sample 
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Table  J-2.  Correlations  Among  the  Dependent  Measures  by  Operator  Type 


-.117  -.073  -.108  -.291*  -.127  -;224  -.177 


